aashish1904 commited on
Commit
bf0565c
1 Parent(s): c744691

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +231 -0
README.md ADDED
@@ -0,0 +1,231 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+
4
+ license: apache-2.0
5
+ language:
6
+ - en
7
+ - fr
8
+ - de
9
+ - es
10
+ - it
11
+ - pt
12
+ - ru
13
+ - zh
14
+ - ja
15
+
16
+ ---
17
+
18
+ ![](https://cdn.discordapp.com/attachments/791342238541152306/1264099835221381251/image.png?ex=669ca436&is=669b52b6&hm=129f56187c31e1ed22cbd1bcdbc677a2baeea5090761d2f1a458c8b1ec7cca4b&)
19
+
20
+ # QuantFactory/Mistral-Nemo-Instruct-2407-GGUF-iamlemec
21
+ This is quantized version of [mistralai/Mistral-Nemo-Instruct-2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) created using llama.cpp
22
+
23
+ # Original Model Card
24
+
25
+
26
+ # Model Card for Mistral-Nemo-Instruct-2407
27
+
28
+ The Mistral-Nemo-Instruct-2407 Large Language Model (LLM) is an instruct fine-tuned version of the [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407). Trained jointly by Mistral AI and NVIDIA, it significantly outperforms existing models smaller or similar in size.
29
+
30
+ For more details about this model please refer to our release [blog post](https://mistral.ai/news/mistral-nemo/).
31
+
32
+ ## Key features
33
+ - Released under the **Apache 2 License**
34
+ - Pre-trained and instructed versions
35
+ - Trained with a **128k context window**
36
+ - Trained on a large proportion of **multilingual and code data**
37
+ - Drop-in replacement of Mistral 7B
38
+
39
+ ## Model Architecture
40
+ Mistral Nemo is a transformer model, with the following architecture choices:
41
+ - **Layers:** 40
42
+ - **Dim:** 5,120
43
+ - **Head dim:** 128
44
+ - **Hidden dim:** 14,436
45
+ - **Activation Function:** SwiGLU
46
+ - **Number of heads:** 32
47
+ - **Number of kv-heads:** 8 (GQA)
48
+ - **Vocabulary size:** 2**17 ~= 128k
49
+ - **Rotary embeddings (theta = 1M)**
50
+
51
+ ## Metrics
52
+
53
+ ### Main Benchmarks
54
+
55
+ | Benchmark | Score |
56
+ | --- | --- |
57
+ | HellaSwag (0-shot) | 83.5% |
58
+ | Winogrande (0-shot) | 76.8% |
59
+ | OpenBookQA (0-shot) | 60.6% |
60
+ | CommonSenseQA (0-shot) | 70.4% |
61
+ | TruthfulQA (0-shot) | 50.3% |
62
+ | MMLU (5-shot) | 68.0% |
63
+ | TriviaQA (5-shot) | 73.8% |
64
+ | NaturalQuestions (5-shot) | 31.2% |
65
+
66
+ ### Multilingual Benchmarks (MMLU)
67
+
68
+ | Language | Score |
69
+ | --- | --- |
70
+ | French | 62.3% |
71
+ | German | 62.7% |
72
+ | Spanish | 64.6% |
73
+ | Italian | 61.3% |
74
+ | Portuguese | 63.3% |
75
+ | Russian | 59.2% |
76
+ | Chinese | 59.0% |
77
+ | Japanese | 59.0% |
78
+
79
+ ## Usage
80
+
81
+ The model can be used with three different frameworks
82
+
83
+ - [`mistral_inference`](https://github.com/mistralai/mistral-inference): See [here](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407#mistral-inference)
84
+ - [`transformers`](https://github.com/huggingface/transformers): See [here](#transformers)
85
+ - [`NeMo`](https://github.com/NVIDIA/NeMo): See [nvidia/Mistral-NeMo-12B-Instruct](https://huggingface.co/nvidia/Mistral-NeMo-12B-Instruct)
86
+
87
+ ### Mistral Inference
88
+
89
+ #### Install
90
+
91
+ It is recommended to use `mistralai/Mistral-Nemo-Instruct-2407` with [mistral-inference](https://github.com/mistralai/mistral-inference). For HF transformers code snippets, please keep scrolling.
92
+
93
+ ```
94
+ pip install mistral_inference
95
+ ```
96
+
97
+ #### Download
98
+
99
+ ```py
100
+ from huggingface_hub import snapshot_download
101
+ from pathlib import Path
102
+
103
+ mistral_models_path = Path.home().joinpath('mistral_models', 'Nemo-Instruct')
104
+ mistral_models_path.mkdir(parents=True, exist_ok=True)
105
+
106
+ snapshot_download(repo_id="mistralai/Mistral-Nemo-Instruct-2407", allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"], local_dir=mistral_models_path)
107
+ ```
108
+
109
+ #### Chat
110
+
111
+ After installing `mistral_inference`, a `mistral-chat` CLI command should be available in your environment. You can chat with the model using
112
+
113
+ ```
114
+ mistral-chat $HOME/mistral_models/Nemo-Instruct --instruct --max_tokens 256 --temperature 0.35
115
+ ```
116
+
117
+ *E.g.* Try out something like:
118
+ ```
119
+ How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar.
120
+ ```
121
+
122
+ #### Instruct following
123
+
124
+ ```py
125
+ from mistral_inference.transformer import Transformer
126
+ from mistral_inference.generate import generate
127
+
128
+ from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
129
+ from mistral_common.protocol.instruct.messages import UserMessage
130
+ from mistral_common.protocol.instruct.request import ChatCompletionRequest
131
+
132
+ tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tekken.json")
133
+ model = Transformer.from_folder(mistral_models_path)
134
+
135
+ prompt = "How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar."
136
+
137
+ completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])
138
+
139
+ tokens = tokenizer.encode_chat_completion(completion_request).tokens
140
+
141
+ out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
142
+ result = tokenizer.decode(out_tokens[0])
143
+
144
+ print(result)
145
+ ```
146
+
147
+ #### Function calling
148
+
149
+ ```py
150
+ from mistral_common.protocol.instruct.tool_calls import Function, Tool
151
+ from mistral_inference.transformer import Transformer
152
+ from mistral_inference.generate import generate
153
+
154
+ from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
155
+ from mistral_common.protocol.instruct.messages import UserMessage
156
+ from mistral_common.protocol.instruct.request import ChatCompletionRequest
157
+
158
+
159
+ tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tekken.json")
160
+ model = Transformer.from_folder(mistral_models_path)
161
+
162
+ completion_request = ChatCompletionRequest(
163
+ tools=[
164
+ Tool(
165
+ function=Function(
166
+ name="get_current_weather",
167
+ description="Get the current weather",
168
+ parameters={
169
+ "type": "object",
170
+ "properties": {
171
+ "location": {
172
+ "type": "string",
173
+ "description": "The city and state, e.g. San Francisco, CA",
174
+ },
175
+ "format": {
176
+ "type": "string",
177
+ "enum": ["celsius", "fahrenheit"],
178
+ "description": "The temperature unit to use. Infer this from the users location.",
179
+ },
180
+ },
181
+ "required": ["location", "format"],
182
+ },
183
+ )
184
+ )
185
+ ],
186
+ messages=[
187
+ UserMessage(content="What's the weather like today in Paris?"),
188
+ ],
189
+ )
190
+
191
+ tokens = tokenizer.encode_chat_completion(completion_request).tokens
192
+
193
+ out_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
194
+ result = tokenizer.decode(out_tokens[0])
195
+
196
+ print(result)
197
+ ```
198
+
199
+ ### Transformers
200
+
201
+ > [!IMPORTANT]
202
+ > NOTE: Until a new release has been made, you need to install transformers from source:
203
+ > ```sh
204
+ > pip install git+https://github.com/huggingface/transformers.git
205
+ > ```
206
+
207
+ If you want to use Hugging Face `transformers` to generate text, you can do something like this.
208
+
209
+ ```py
210
+ from transformers import pipeline
211
+
212
+ messages = [
213
+ {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
214
+ {"role": "user", "content": "Who are you?"},
215
+ ]
216
+ chatbot = pipeline("text-generation", model="mistralai/Mistral-Nemo-Instruct-2407")
217
+ chatbot(messages)
218
+ ```
219
+
220
+ > [!TIP]
221
+ > Unlike previous Mistral models, Mistral Nemo requires smaller temperatures. We recommend to use a temperature of 0.3.
222
+
223
+ ## Limitations
224
+
225
+ The Mistral Nemo Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance.
226
+ It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to
227
+ make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.
228
+
229
+ ## The Mistral AI Team
230
+
231
+ Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Alok Kothari, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Augustin Garreau, Austin Birky, Bam4d, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Carole Rambaud, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gaspard Blanchet, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Hichem Sattouf, Ian Mack, Jean-Malo Delignon, Jessica Chudnovsky, Justus Murke, Kartik Khandelwal, Lawrence Stewart, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Marjorie Janiewicz, Mickaël Seznec, Nicolas Schuhl, Niklas Muhs, Olivier de Garrigues, Patrick von Platen, Paul Jacob, Pauline Buche, Pavan Kumar Reddy, Perry Savas, Pierre Stock, Romain Sauvestre, Sagar Vaze, Sandeep Subramanian, Saurabh Garg, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibault Schueller, Thibaut Lavril, Thomas Wang, Théophile Gervet, Timothée Lacroix, Valera Nemychnikova, Wendy Shang, William El Sayed, William Marshall