---
language:
- en
library_name: peft
pipeline_tag: text-generation
license: llama2
datasets:
- emrgnt-cmplxty/sciphi-textbooks-are-all-you-need
---
# ML1 Previews
This repository contains the previews for the ML1 model - [Reddit Post](https://www.reddit.com/r/LocalLLaMA/comments/16ul4sw/ml1_34b70b_phi_115_reproduction_on_llama2/)
Watch training live here: [https://api.wandb.ai/links/nickmitchko/t5d47kzr](https://api.wandb.ai/links/nickmitchko/t5d47kzr)
## Checkpoints
| Model | 1 Epoch Pct | Link |
|---------------|--------|-------|
| ML1-34b | 15% | [Directory](https://huggingface.co./nmitchko/ML1-34b-previews/tree/main/checkpoint-1) |
| ML1-34b | 50% | ~ |
| ML1-34b | 100% | ~ |
| ML1-mistral-7b| 50% | ~ |
| ML1-mistral-7b| 100%|~|
| ML1-70b | 15% | ~ |
| ML1-70b | 50% | ~ |
| ML1-70b | 100% | ~ |
## Model Description
The goal is to develop a series of models that can express superior performance given high quality data. To achieve this, I plan to experiment with the lovely dataset produced by [/u/docsoc1](https://www.reddit.com/user/docsoc1). Huge shout out to him/her! If you'd like to view that dataset, the link is below.
Dataset: [emrgnt-cmplxty/sciphi-textbooks-are-all-you-need](https://huggingface.co./datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
## Prompt Format
The model is trained using the alpaca format. Please see [here](https://github.com/tatsu-lab/stanford_alpaca#data-release) or below for that format:
```text
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
```
### Architecture
`nmitchko/ML1-34b-previews` is a large language model repository of LoRA checkpoints specifically fine-tuned to add text-book synthesized data in the style of Phi 1/1.5.
It is based on [`codellama-34b-hf`](https://huggingface.co./codellama/CodeLlama-34b-hf) at 34 billion parameters.
The primary goal of this model is to test various fine tuning methods around high quality data.
It was trained using [LoRA](https://arxiv.org/abs/2106.09685), specifically [QLora Multi GPU](https://github.com/ChrisHayduk/qlora-multi-gpu), to reduce memory footprint.
See Training Parameters for more info This Lora supports 4-bit and 8-bit modes.
### Requirements
```
bitsandbytes>=0.41.0
peft@main
transformers@main
```
Steps to load this model:
1. Load base model (codellama-34b-hf) using transformers
2. Download a checkpoint folder (checkpoint-1)
3. Apply LoRA using peft
## Training Parameters
The model is currently training on [emrgnt-cmplxty/sciphi-textbooks-are-all-you-need](https://huggingface.co./datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
`emrgnt-cmplxty/sciphi-textbooks-are-all-you-need` contains textbook synthesized data.
| Item | Amount | Units |
|---------------|--------|-------|
| LoRA Rank | 64 | ~ |
| LoRA Alpha | 16 | ~ |
| Learning Rate | 1e-4 | SI |
| Dropout | 5 | % |
## Training procedure
The following `bitsandbytes` quantization config was used during training:
- quant_method: QuantizationMethod.BITS_AND_BYTES
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
### Framework versions
- PEFT 0.6.0.dev0