yash21 commited on
Commit
4c212c0
1 Parent(s): 1448d4d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -46
README.md CHANGED
@@ -1,60 +1,16 @@
1
  ---
2
- license: apache-2.0
3
  tags:
4
- - moe
5
- - merge
6
- - mergekit
7
- - lazymergekit
8
- - OpenPipe/mistral-ft-optimized-1227
9
- - openchat/openchat-3.5-1210
10
- - HuggingFaceH4/zephyr-7b-beta
11
- - meta-math/MetaMath-Mistral-7B
12
  ---
13
 
14
  # OpenMistral-MoE
15
 
16
- OpenMistral-MoE is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
17
  * [OpenPipe/mistral-ft-optimized-1227](https://huggingface.co/OpenPipe/mistral-ft-optimized-1227)
18
  * [openchat/openchat-3.5-1210](https://huggingface.co/openchat/openchat-3.5-1210)
19
  * [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)
20
  * [meta-math/MetaMath-Mistral-7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B)
21
 
22
- ## 🧩 Configuration
23
-
24
- ```yaml
25
- base_model: mistralai/Mistral-7B-Instruct-v0.2
26
- gate_mode: hidden
27
- dtype: bfloat16
28
- merge_method: dare_ties
29
- experts:
30
- - source_model: OpenPipe/mistral-ft-optimized-1227
31
- positive_prompts:
32
- - "chat"
33
- - "assistant"
34
- - "tell me"
35
- - "explain"
36
- - source_model: openchat/openchat-3.5-1210
37
- positive_prompts:
38
- - "code"
39
- - "python"
40
- - "javascript"
41
- - "programming"
42
- - "algorithm"
43
- - source_model: HuggingFaceH4/zephyr-7b-beta
44
- positive_prompts:
45
- - "storywriting"
46
- - "write"
47
- - "scene"
48
- - "story"
49
- - "character"
50
- - source_model: meta-math/MetaMath-Mistral-7B
51
- positive_prompts:
52
- - "reason"
53
- - "math"
54
- - "mathematics"
55
- - "solve"
56
- - "count"
57
- ```
58
 
59
  ## 💻 Usage
60
 
 
1
  ---
 
2
  tags:
3
+ - MoE
 
 
 
 
 
 
 
4
  ---
5
 
6
  # OpenMistral-MoE
7
 
8
+ OpenMistral-MoE is a Mixure of Experts (MoE)
9
  * [OpenPipe/mistral-ft-optimized-1227](https://huggingface.co/OpenPipe/mistral-ft-optimized-1227)
10
  * [openchat/openchat-3.5-1210](https://huggingface.co/openchat/openchat-3.5-1210)
11
  * [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)
12
  * [meta-math/MetaMath-Mistral-7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B)
13
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
 
15
  ## 💻 Usage
16