TheBloke commited on
Commit
284e405
1 Parent(s): 8091280

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -103
README.md CHANGED
@@ -38,11 +38,18 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
38
  * [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
39
  * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
40
 
 
 
 
 
 
 
 
 
41
  ## Repositories available
42
 
43
  * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/NewHope-GPTQ)
44
  * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/NewHope-GGML)
45
- * [SLAM-group's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/SLAM-group/NewHope)
46
 
47
  ## Prompt template: Alpaca
48
 
@@ -153,105 +160,4 @@ Thank you to all my generous patrons and donaters!
153
 
154
  # Original model card: SLAM-group's NewHope
155
 
156
- # NewHope: Harnessing 99% of GPT-4's Programming Capabilities
157
-
158
- We introduce NewHope, a fine-tuned chat model based on llama-2-13b, aiming to provide a strong coding capability. NewHope handle different languages including Python, C++, Java, JavaScript, Go, and more. Preliminary evaluation on HumanEval shows that **NewHope possesses 99% of GPT-4's programming capabilities**.
159
-
160
- **Contact**: SLAM (<ins>S</ins>UFE <ins>L</ins>arge <ins>A</ins>I <ins>M</ins>odel) is a research group at Shanghai University of Finance and Economics.
161
162
-
163
- **TODO**: We will release more evaluatation results and training details later.
164
-
165
- # Evaluation Results
166
-
167
- We evaluated NewHope on [HumanEval](https://github.com/openai/human-eval) using the official evaluation script by OpenAI. We compared the Pass@1 metric of NewHope with other models. The results of other models are from PapersWithCode.
168
-
169
- | Model | Pass@1 |
170
- | ----- | ------ |
171
- | **GPT-4** | **67.0** |
172
- | **NewHope** | **66.5** |
173
- | PanGu-Coder2 15B | 61.6 |
174
- | WizardCoder 15B | 57.3 |
175
- | phi-1 1.3B | 50.6 |
176
- | GPT-3.5 | 48.1 |
177
- | phi-1-small | 45.0 |
178
- | PaLM-Coder | 36.0 |
179
- | CodeGeeX2-6B | 35.9 |
180
-
181
- # Model Weights
182
-
183
- We have open-sourced the model weights [NewHope](https://huggingface.co/SLAM-group/NewHope).
184
-
185
- We are uploading the model weights. The weights will be available in a few hours.
186
-
187
-
188
- # Usage
189
-
190
- To load the NewHope model using Transformers, use the following code:
191
- ```
192
- import torch
193
- from transformers import LlamaTokenizer, LlamaForCausalLM
194
-
195
- base_model = "SLAM-group/NewHope"
196
- tokenizer = LlamaTokenizer.from_pretrained(base_model)
197
- model = LlamaForCausalLM.from_pretrained(base_model, torch_dtype=torch.float16, device_map="auto")
198
- # model.config.use_cache is default to `False`. For inference: `model.config.use_cache = True`
199
- ```
200
- **Note:** At least Huggingface Transformers **4.31.0** is required to load this model!
201
-
202
- You can ask NewHope to generate code with instructions. We provide a simple example of how NewHope model generates code with the specific prompt:
203
- ```
204
- # Suppose required tokenizer and model have already been loaded
205
-
206
- instruction = "Write a Python function to tell me what the date is today."
207
- prompt = f"<s> ### Instruction:\n{instruction}\n\n### Response:\n"
208
- inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").to("cuda")
209
- output = model.generate(**inputs, do_sample=True, top_p=0.9, max_new_tokens=2048)[0]
210
- decoded_output = tokenizer.decode(output, skip_special_tokens=True).split("### Response:\n")[-1].strip()
211
- print(decoded_output)
212
- ```
213
-
214
- You can also interact with NewHope in a dialog manner with the following prompt:
215
- ```
216
- <s> ### Instruction:\nQ1\n\n### Response:\nA1</s><s> ### Instruction:\nQ2\n\n### Response:\nA2</s>
217
- ```
218
-
219
-
220
- # Evaluation
221
-
222
- ### Local setup
223
- 1. Install HumanEval for evaluation. [Details](https://github.com/openai/human-eval)
224
- 2. Install dependencies
225
-
226
- ```bash
227
- pip install -r requirements.txt
228
- ```
229
-
230
- ---
231
- For HumanEval, we use the following prompt:
232
- ```
233
- example_input = 'def is_odd(number: int) -> bool:\n """ Check whether the given number is odd\n >>> is_odd(3)\n True\n >>> is_odd(6)\n False\n """\n'
234
- example_output = 'def is_odd(number: int) -> bool:\n """ Check whether the given number is odd\n >>> is_odd(3)\n True\n >>> is_odd(6)\n False\n """\n return number % 2 == 1'
235
-
236
- task_in_humaneval = "REPLACE `task_in_humaneval` WITH THE SPECIFIC TASK IN HUMANEVAL DATA"
237
-
238
- prompt = f"<s> ### Instruction:\nComplete the given function below:\n\n{example_input}\n\n### Response:\n{example_output}</s><s> ### Instruction:\nComplete the given function below:\n\n{task_in_human_eval}\n\n### Response:\n"
239
- ```
240
-
241
- To reproduce the results on HumanEval, use the following script:
242
- ```
243
- python complete.py --base_model SLAM-group/NewHope --output_dir output --n_gpu 8
244
- ```
245
- The above script will generate `samples.jsonl` in `output_dir`, which can be directly evaluated by HumanEval. [Evaluation procedure](https://github.com/openai/human-eval). We conducted the experiment with `fp16` on 8xA800, 80GB GPUs, reaching `66.5%` on Pass@1 (v.s. GPT4 `67.0%`).
246
-
247
- # Citation
248
-
249
- ```
250
- @misc{2023newhope,
251
- title={NewHope: Harnessing 99% of GPT-4's Programming Capabilities},
252
- author={Wanyun Cui and Qianle Wang},
253
- howpublished = https://github.com/SLAM-group/newhope,
254
- year={2023}
255
- }
256
- ```
257
-
 
38
  * [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
39
  * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
40
 
41
+ ### Original model removed
42
+
43
+ The original model creator has removed their model, due to their training dataset being contanimated with samples from the HumanEval dataset.
44
+
45
+ This does not affect the ability of this model, which has proven to be very high quality.
46
+
47
+ For the moment I am going to leave this model up, but will remove it this is requested by the original model creator.
48
+
49
  ## Repositories available
50
 
51
  * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/NewHope-GPTQ)
52
  * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/NewHope-GGML)
 
53
 
54
  ## Prompt template: Alpaca
55
 
 
160
 
161
  # Original model card: SLAM-group's NewHope
162
 
163
+ Original model was removed.