Update README.md
Browse files
README.md
CHANGED
@@ -38,11 +38,18 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
|
|
38 |
* [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
|
39 |
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
|
40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
41 |
## Repositories available
|
42 |
|
43 |
* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/NewHope-GPTQ)
|
44 |
* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/NewHope-GGML)
|
45 |
-
* [SLAM-group's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/SLAM-group/NewHope)
|
46 |
|
47 |
## Prompt template: Alpaca
|
48 |
|
@@ -153,105 +160,4 @@ Thank you to all my generous patrons and donaters!
|
|
153 |
|
154 |
# Original model card: SLAM-group's NewHope
|
155 |
|
156 |
-
|
157 |
-
|
158 |
-
We introduce NewHope, a fine-tuned chat model based on llama-2-13b, aiming to provide a strong coding capability. NewHope handle different languages including Python, C++, Java, JavaScript, Go, and more. Preliminary evaluation on HumanEval shows that **NewHope possesses 99% of GPT-4's programming capabilities**.
|
159 |
-
|
160 |
-
**Contact**: SLAM (<ins>S</ins>UFE <ins>L</ins>arge <ins>A</ins>I <ins>M</ins>odel) is a research group at Shanghai University of Finance and Economics.
|
161 | |
162 |
-
|
163 |
-
**TODO**: We will release more evaluatation results and training details later.
|
164 |
-
|
165 |
-
# Evaluation Results
|
166 |
-
|
167 |
-
We evaluated NewHope on [HumanEval](https://github.com/openai/human-eval) using the official evaluation script by OpenAI. We compared the Pass@1 metric of NewHope with other models. The results of other models are from PapersWithCode.
|
168 |
-
|
169 |
-
| Model | Pass@1 |
|
170 |
-
| ----- | ------ |
|
171 |
-
| **GPT-4** | **67.0** |
|
172 |
-
| **NewHope** | **66.5** |
|
173 |
-
| PanGu-Coder2 15B | 61.6 |
|
174 |
-
| WizardCoder 15B | 57.3 |
|
175 |
-
| phi-1 1.3B | 50.6 |
|
176 |
-
| GPT-3.5 | 48.1 |
|
177 |
-
| phi-1-small | 45.0 |
|
178 |
-
| PaLM-Coder | 36.0 |
|
179 |
-
| CodeGeeX2-6B | 35.9 |
|
180 |
-
|
181 |
-
# Model Weights
|
182 |
-
|
183 |
-
We have open-sourced the model weights [NewHope](https://huggingface.co/SLAM-group/NewHope).
|
184 |
-
|
185 |
-
We are uploading the model weights. The weights will be available in a few hours.
|
186 |
-
|
187 |
-
|
188 |
-
# Usage
|
189 |
-
|
190 |
-
To load the NewHope model using Transformers, use the following code:
|
191 |
-
```
|
192 |
-
import torch
|
193 |
-
from transformers import LlamaTokenizer, LlamaForCausalLM
|
194 |
-
|
195 |
-
base_model = "SLAM-group/NewHope"
|
196 |
-
tokenizer = LlamaTokenizer.from_pretrained(base_model)
|
197 |
-
model = LlamaForCausalLM.from_pretrained(base_model, torch_dtype=torch.float16, device_map="auto")
|
198 |
-
# model.config.use_cache is default to `False`. For inference: `model.config.use_cache = True`
|
199 |
-
```
|
200 |
-
**Note:** At least Huggingface Transformers **4.31.0** is required to load this model!
|
201 |
-
|
202 |
-
You can ask NewHope to generate code with instructions. We provide a simple example of how NewHope model generates code with the specific prompt:
|
203 |
-
```
|
204 |
-
# Suppose required tokenizer and model have already been loaded
|
205 |
-
|
206 |
-
instruction = "Write a Python function to tell me what the date is today."
|
207 |
-
prompt = f"<s> ### Instruction:\n{instruction}\n\n### Response:\n"
|
208 |
-
inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").to("cuda")
|
209 |
-
output = model.generate(**inputs, do_sample=True, top_p=0.9, max_new_tokens=2048)[0]
|
210 |
-
decoded_output = tokenizer.decode(output, skip_special_tokens=True).split("### Response:\n")[-1].strip()
|
211 |
-
print(decoded_output)
|
212 |
-
```
|
213 |
-
|
214 |
-
You can also interact with NewHope in a dialog manner with the following prompt:
|
215 |
-
```
|
216 |
-
<s> ### Instruction:\nQ1\n\n### Response:\nA1</s><s> ### Instruction:\nQ2\n\n### Response:\nA2</s>
|
217 |
-
```
|
218 |
-
|
219 |
-
|
220 |
-
# Evaluation
|
221 |
-
|
222 |
-
### Local setup
|
223 |
-
1. Install HumanEval for evaluation. [Details](https://github.com/openai/human-eval)
|
224 |
-
2. Install dependencies
|
225 |
-
|
226 |
-
```bash
|
227 |
-
pip install -r requirements.txt
|
228 |
-
```
|
229 |
-
|
230 |
-
---
|
231 |
-
For HumanEval, we use the following prompt:
|
232 |
-
```
|
233 |
-
example_input = 'def is_odd(number: int) -> bool:\n """ Check whether the given number is odd\n >>> is_odd(3)\n True\n >>> is_odd(6)\n False\n """\n'
|
234 |
-
example_output = 'def is_odd(number: int) -> bool:\n """ Check whether the given number is odd\n >>> is_odd(3)\n True\n >>> is_odd(6)\n False\n """\n return number % 2 == 1'
|
235 |
-
|
236 |
-
task_in_humaneval = "REPLACE `task_in_humaneval` WITH THE SPECIFIC TASK IN HUMANEVAL DATA"
|
237 |
-
|
238 |
-
prompt = f"<s> ### Instruction:\nComplete the given function below:\n\n{example_input}\n\n### Response:\n{example_output}</s><s> ### Instruction:\nComplete the given function below:\n\n{task_in_human_eval}\n\n### Response:\n"
|
239 |
-
```
|
240 |
-
|
241 |
-
To reproduce the results on HumanEval, use the following script:
|
242 |
-
```
|
243 |
-
python complete.py --base_model SLAM-group/NewHope --output_dir output --n_gpu 8
|
244 |
-
```
|
245 |
-
The above script will generate `samples.jsonl` in `output_dir`, which can be directly evaluated by HumanEval. [Evaluation procedure](https://github.com/openai/human-eval). We conducted the experiment with `fp16` on 8xA800, 80GB GPUs, reaching `66.5%` on Pass@1 (v.s. GPT4 `67.0%`).
|
246 |
-
|
247 |
-
# Citation
|
248 |
-
|
249 |
-
```
|
250 |
-
@misc{2023newhope,
|
251 |
-
title={NewHope: Harnessing 99% of GPT-4's Programming Capabilities},
|
252 |
-
author={Wanyun Cui and Qianle Wang},
|
253 |
-
howpublished = https://github.com/SLAM-group/newhope,
|
254 |
-
year={2023}
|
255 |
-
}
|
256 |
-
```
|
257 |
-
|
|
|
38 |
* [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
|
39 |
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
|
40 |
|
41 |
+
### Original model removed
|
42 |
+
|
43 |
+
The original model creator has removed their model, due to their training dataset being contanimated with samples from the HumanEval dataset.
|
44 |
+
|
45 |
+
This does not affect the ability of this model, which has proven to be very high quality.
|
46 |
+
|
47 |
+
For the moment I am going to leave this model up, but will remove it this is requested by the original model creator.
|
48 |
+
|
49 |
## Repositories available
|
50 |
|
51 |
* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/NewHope-GPTQ)
|
52 |
* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/NewHope-GGML)
|
|
|
53 |
|
54 |
## Prompt template: Alpaca
|
55 |
|
|
|
160 |
|
161 |
# Original model card: SLAM-group's NewHope
|
162 |
|
163 |
+
Original model was removed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|