File size: 5,583 Bytes

---
license: cc-by-nc-4.0
datasets:
- conceptofmind/cot_submix_original
- conceptofmind/flan2021_submix_original
- conceptofmind/t0_submix_original
- conceptofmind/niv2_submix_original
language:
- en
pipeline_tag: text-generation
---
# Stable Belgua 1

## Model Description

`Stable Beluga 1` is a Llama65B model fine-tuned on an Orca style Dataset

## Usage

### Apply Delta Weights

Stable Beluga 1 cannot be used from the `stabilityai/StableBeluga1-Delta` weights alone. To obtain the correct model, one must add back the difference between LLaMA 65B and `stabilityai/StableBeluga1-Delta` weights. We provide the [`apply_delta.py`](https://huggingface.co./stabilityai/StabelBeluga1-Delta/raw/main/apply_delta.py) script to automate the conversion, which you can run as:


```sh
python3 apply_delta.py --base-model-path /path/to/model_weights/llama-65b --target-model-path StableBeluga1 --delta-path stabilityai/StableBeluga1-Delta
```



Start chatting with `Stable Beluga 1` using the following code snippet:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("your_path_to_StableBeluga1", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("your_path_to_StableBeluga1", torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="auto")

system_prompt = "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n"
system_prompt += "### Instruction:\nYou are Stable Beluga, an AI that follows instructions extremely well. Help as much as you can. Remember, be safe, and don't do anything illegal.\n\n"

message = "Write me a poem please"
prompt = f"{system_prompt}### Input: {message}\n\n### Response:\n"

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=256)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

Stable Beluga 1 should be used with prompts formatted similarly to Alpaca as below:
```
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

## Instruction:
This is a system prompt, please behave and help the user.

### Input:
Your prompt here

### Response:
The output of Stable Beluga 1
```

## Model Details

* **Developed by**: [Stability AI](https://stability.ai/)
* **Model type**: Stable Beluga 1 is an auto-regressive language model fine-tuned on LLaMA65B.
* **Language(s)**: English
* **Library**: [HuggingFace Transformers](https://github.com/huggingface/transformers)
* **License**: Fine-tuned checkpoints (`StableBeluga1`) is licensed under the Non-Commercial Creative Commons license ([CC BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/))
* **Contact**: For questions and comments about the model, please email `[email protected]`

### Training Dataset

`Stable Beluga 1` is trained on our internal Orca-style dataset

### Training Procedure

Models are learned via supervised fine-tuning on the aforementioned datasets, trained in mixed-precision (BF16), and optimized with AdamW. We outline the following hyperparameters:

| Dataset           | Batch Size | Learning Rate |Learning Rate Decay| Warm-up | Weight Decay | Betas       |
|-------------------|------------|---------------|-------------------|---------|--------------|-------------|
| Orca pt1 packed   | 512        | 3e-5          | Cosine to 3e-6    | 100     | 1e-6         | (0.9, 0.95) |
| Orca pt2 unpacked | 512        | 3e-5          | Cosine to 3e-6    | 100     | 1e-6         | (0.9, 0.95) |

## Use and Limitations

### Ethical Considerations and Limitations

Beluga is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Beluga's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Beluga, developers should perform safety testing and tuning tailored to their specific applications of the model.


## Citations

```bibtext
@misc{touvron2023llama,
      title={LLaMA: Open and Efficient Foundation Language Models}, 
      author={Hugo Touvron and Thibaut Lavril and Gautier Izacard and Xavier Martinet and Marie-Anne Lachaux and Timothée Lacroix and Baptiste Rozière and Naman Goyal and Eric Hambro and Faisal Azhar and Aurelien Rodriguez and Armand Joulin and Edouard Grave and Guillaume Lample},
      year={2023},
      eprint={2302.13971},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

```bibtext
@misc{mukherjee2023orca,
      title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4}, 
      author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
      year={2023},
      eprint={2306.02707},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

```bibtex
@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
```