Llama-3.1-8B-Instruct-EI1-120K-32gpus-2ep-sft

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4598

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6e-06
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 32
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• total_eval_batch_size: 256
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0562	100	0.6314
No log	0.1124	200	0.5978
No log	0.1685	300	0.5793
No log	0.2247	400	0.5625
0.6013	0.2809	500	0.5482
0.6013	0.3371	600	0.5350
0.6013	0.3933	700	0.5237
0.6013	0.4494	800	0.5150
0.6013	0.5056	900	0.5074
0.5076	0.5618	1000	0.5004
0.5076	0.6180	1100	0.4940
0.5076	0.6742	1200	0.4897
0.5076	0.7303	1300	0.4853
0.5076	0.7865	1400	0.4813
0.4762	0.8427	1500	0.4776
0.4762	0.8989	1600	0.4751
0.4762	0.9551	1700	0.4719
0.4762	1.0112	1800	0.4737
0.4762	1.0674	1900	0.4719
0.4385	1.1236	2000	0.4713
0.4385	1.1798	2100	0.4701
0.4385	1.2360	2200	0.4684
0.4385	1.2921	2300	0.4674
0.4385	1.3483	2400	0.4663
0.4025	1.4045	2500	0.4650
0.4025	1.4607	2600	0.4634
0.4025	1.5169	2700	0.4630
0.4025	1.5730	2800	0.4623
0.4025	1.6292	2900	0.4613
0.3979	1.6854	3000	0.4608
0.3979	1.7416	3100	0.4605
0.3979	1.7978	3200	0.4602
0.3979	1.8539	3300	0.4599
0.3979	1.9101	3400	0.4598
0.3972	1.9663	3500	0.4598

Framework versions

• Transformers 4.43.4
• Pytorch 2.4.0+cu121
• Datasets 3.0.1
• Tokenizers 0.19.1