Llama-3.1-8B-Instruct-dpo-1000

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the bct_non_cot_dpo_1000 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2845
• Rewards/chosen: 0.1535
• Rewards/rejected: -1.8119
• Rewards/accuracies: 0.9000
• Rewards/margins: 1.9654
• Logps/chosen: -16.0986
• Logps/rejected: -38.2013
• Logits/chosen: -0.1852
• Logits/rejected: -0.3689

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/chosen	Logps/rejected	Logits/chosen	Logits/rejected
0.6346	1.7778	50	0.6252	0.2207	0.0694	0.8000	0.1513	-15.4268	-19.3885	-0.2672	-0.3872
0.4162	3.5556	100	0.3736	0.3857	-0.7267	0.8900	1.1125	-13.7764	-27.3496	-0.2058	-0.3821
0.2919	5.3333	150	0.3053	0.2701	-1.3825	0.9000	1.6526	-14.9327	-33.9072	-0.1906	-0.3753
0.3007	7.1111	200	0.2881	0.1886	-1.7019	0.9100	1.8905	-15.7478	-37.1016	-0.1883	-0.3726
0.2536	8.8889	250	0.2845	0.1535	-1.8119	0.9000	1.9654	-16.0986	-38.2013	-0.1852	-0.3689

Framework versions

• PEFT 0.12.0
• Transformers 4.45.2
• Pytorch 2.3.0
• Datasets 2.19.0
• Tokenizers 0.20.0