README.md

metadata

base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-3
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-3_60-samples_config-2_full
    results: []

Llama-31-8B_task-3_60-samples_config-2_full

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-3 dataset. It achieves the following results on the evaluation set:

• Loss: 1.1641

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss
1.6831	0.6957	2	1.6900
1.6856	1.7391	5	1.6448
1.6065	2.7826	8	1.5643
1.5184	3.8261	11	1.4830
1.4227	4.8696	14	1.4134
1.3635	5.9130	17	1.3473
1.2487	6.9565	20	1.2794
1.2196	8.0	23	1.2358
1.1686	8.6957	25	1.2184
1.1271	9.7391	28	1.2041
1.1329	10.7826	31	1.1925
1.1049	11.8261	34	1.1838
1.067	12.8696	37	1.1764
1.0693	13.9130	40	1.1707
1.039	14.9565	43	1.1672
1.0381	16.0	46	1.1651
0.994	16.6957	48	1.1641
1.0091	17.7391	51	1.1648
0.996	18.7826	54	1.1667
0.969	19.8261	57	1.1695
0.9577	20.8696	60	1.1710
0.9489	21.9130	63	1.1720
0.9253	22.9565	66	1.1765
0.9133	24.0	69	1.1808

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1