update readme

README.md

@@ -6,7 +6,7 @@ license: cc-by-nc-4.0
 
 | [Paper](https://arxiv.org/abs/2305.14314) | [Code](https://github.com/artidoro/qlora) |
 
-**The `LLaMA-2 QLoRA OpenOrca` are open-source models obtained through 4-bit QLoRA tuning of LLaMA-2 base models 240k exmaples of OpenOrca.**
+**The `LLaMA-2 QLoRA OpenOrca` are open-source models obtained through 4-bit QLoRA tuning of LLaMA-2 base models 240k exmaples of [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca).**
 
 ⚠️ These models are purely intended for research purposes and could produce problematic outputs. 
 
@@ -17,7 +17,7 @@ license: cc-by-nc-4.0
 - **Lightweight** checkpoints which only contain adapter weights.
 
 ## License and Intended Use
-Note the use of these adapter weights, requires access to the LLaMA-2 model weighs and therefore should be used according to the LLaMA-2 license. 
+Note the use of these adapter weights, requires access to the LLaMA-2 model weighs and therefore should be used according to the LLaMA-2 license. The adapter weights are trained on data obtained from OpenAI GPT-3.5 and GPT-4 models (see more details in the Finetuning Data section). As such any use of these adapters should follow their license.
 
 ## Usage
 Here is an example of how you would load the model 4-bits:
@@ -47,28 +47,21 @@ tokenizer = AutoTokenizer.from_pretrained(model_name)
 ```
 Inference can then be performed as usual with HF models as follows:
 ```python
-prompt = "Introduce yourself"
+question = "Explain Einstein's theory of special relativity."
 formatted_prompt = (
-    f"A chat between a curious human and an artificial intelligence assistant."
-    f"The assistant gives helpful, detailed, and polite answers to the user's questions.\n"
-    f"### Human: {prompt} ### Assistant:"
+    f"### Instruction: {question}\n\n### Response:"
 )
 inputs = tokenizer(formatted_prompt, return_tensors="pt").to("cuda:0")
 outputs = model.generate(inputs=inputs.input_ids, max_new_tokens=20)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
-Expected output similar to the following:
-```
-A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
-### Human: Introduce yourself ### Assistant: I am an artificial intelligence assistant. I am here to help you with any questions you may have.
-```
 
 ## Model Card
 **Architecture**: The models released here are LoRA adapters to be used on top of LLaMA-2 models. They are added to all layers. For all model sizes, we use $r=64$.
 
 **Base Model**: These models use LLaMA-2 as base model. LLaMA is a causal language model pretrained on a large corpus of text. See [LLaMA-2 paper](https://arxiv.org/abs/2307.09288) for more details. Note that these models can inherit biases and limitations of the base model.
 
-**Finetuning Data**: These models are finetuned on 240k examples of the [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset.
+**Finetuning Data**: These models are finetuned on 240k examples of the [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset. The OpenOrca dataset is a replica of the [Orca](https://arxiv.org/abs/2306.02707) dataset which uses FLAN v2 prompts and GPT3.5/4 completions.
 
 
 **Languages**: The different datasets cover different languages. We direct to the various papers and resources describing the datasets for more details. 
@@ -84,13 +77,20 @@ For the finetuning process, we use constant learning rate schedule and paged Ada
 ### Training hyperparameters
 | Parameters | Dataset  | Batch size | LR   | Steps | Source Length | Target Length |
 |------------|----------|------------|------|-------|---------------|---------------|
-| 7B         | All      | 16         | 2e-4 | 10000 | 384           | 128           |
-| 13B        | All      | 16         | 2e-4 | 10000 | 384           | 128           |
-| 70B        | All      | 64         | 1e-4 | 2500  | 384           | 128           |
+| 7B         | OpenOrca      | 16         | 2e-4 | 15000 | 384           | 128           |
+| 13B        | OpenOrca      | 16         | 2e-4 | 15000 | 384           | 128           |
+| 70B        | OpenOrca      | 64         | 1e-4 | 3750  | 384           | 128           |
 
 ### Evaluation
 We use the MMLU benchmark to measure performance on a range of language understanding tasks. This is a multiple-choice benchmark covering 57 tasks including elementary mathematics, US history, computer science, law, and more. We report 5-shot test accuracy.
 
+Dataset | 7B | 13B | 34B | 70B 
+---|---|---|---|---
+ LLaMA-2 no tuning |  45.3 | 54.8 |  62.6 | 68.9
+ OpenOrca | 45.0 |  |  | 69.0
+
+For reference here are the MMLU results of QLoRA finetuning on other datasets:
+
  Dataset | 7B | 13B | 33B | 65B 
 ---|---|---|---|---
  LLaMA-1 no tuning | 35.1 | 46.9 | 57.8 | 63.4 
@@ -103,11 +103,6 @@ We use the MMLU benchmark to measure performance on a range of language understa
  Alpaca | 38.8 | 47.8 | 57.3 | 62.5 
  FLAN v2 | 44.5 | 51.4 | 59.2 | 63.9 
 
-Dataset | 7B | 13B | 34B | 70B 
----|---|---|---|---
- LLaMA-2 no tuning |  45.3 | 54.8 |  62.6 | 68.9
- OpenOrca | 45.0 |  |  | 69.0
-
 
 ## Citation