Mistral 7B fine-tuned on H4/No Robots instructions
This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the HuggingFaceH4/no_robots dataset for instruction following downstream task.
Training procedure
The model was loaded on 8 bits and fine-tuned on the LIMA dataset using the LoRA PEFT technique with the huggingface/peft
library and trl/sft
for one epoch on 1 x A100 (40GB) GPU.
SFT Trainer params:
trainer = SFTTrainer(
model=model,
train_dataset=train_ds,
eval_dataset=test_ds,
peft_config=peft_config,
dataset_text_field="text",
max_seq_length=2048,
tokenizer=tokenizer,
args=training_arguments,
packing=False
)
LoRA config:
config = LoraConfig(
lora_alpha=16,
lora_dropout=0.1,
r=64,
bias="none",
task_type="CAUSAL_LM",
target_modules = ['q_proj', 'k_proj', 'down_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj']
)
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 8
- seed: 66
- gradient_accumulation_steps: 64
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 2
- mixed_precision_training: Native AMP
Training results
Step | Training Loss | Validation Loss |
---|---|---|
10 | 1.796200 | 1.774305 |
20 | 1.769700 | 1.679720 |
30 | 1.626800 | 1.667754 |
40 | 1.663400 | 1.665188 |
50 | 1.565700 | 1.659000 |
60 | 1.660300 | 1.658270 |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
repo_id = "mrm8488/mistral-7b-ft-h4-no_robots_instructions"
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(repo_id)
gen = pipeline("text-generation", model=model, tokenizer=tokenizer, device=0)
instruction = "[INST] Write an email to say goodbye to me boss [\INST]"
res = gen(instruction, max_new_tokens=512, temperature=0.3, top_p=0.75, top_k=40, repetition_penalty=1.2, eos_token_id=2)
print(res[0]['generated_text'])
Framework versions
- Transformers 4.35.0.dev0
- Pytorch 2.1.0+cu118
- Datasets 2.14.6
- Tokenizers 0.14.1
Citation
@misc {manuel_romero_2023,
author = { {Manuel Romero} },
title = { mistral-7b-ft-h4-no_robots_instructions (Revision 785446d) },
year = 2023,
url = { https://huggingface.co./mrm8488/mistral-7b-ft-h4-no_robots_instructions },
doi = { 10.57967/hf/1426 },
publisher = { Hugging Face }
}
- Downloads last month
- 28
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for mrm8488/mistral-7b-ft-h4-no_robots_instructions
Base model
mistralai/Mistral-7B-v0.1