ft-raft-hotpot

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.8450

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 1399
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
1.61	0.0148	20	1.3531
1.1917	0.0295	40	1.0786
1.0438	0.0443	60	1.0227
1.0087	0.0590	80	0.9971
0.9835	0.0738	100	0.9795
0.9753	0.0885	120	0.9651
0.9586	0.1033	140	0.9521
0.9504	0.1180	160	0.9419
0.937	0.1328	180	0.9312
0.9242	0.1475	200	0.9203
0.9073	0.1623	220	0.9111
0.9017	0.1771	240	0.9023
0.8956	0.1918	260	0.8939
0.8824	0.2066	280	0.8856
0.8776	0.2213	300	0.8784
0.8755	0.2361	320	0.8710
0.8746	0.2508	340	0.8646
0.8569	0.2656	360	0.8592
0.8499	0.2803	380	0.8547
0.8514	0.2951	400	0.8509
0.8541	0.3098	420	0.8484
0.856	0.3246	440	0.8465
0.8397	0.3394	460	0.8455
0.8402	0.3541	480	0.8451
0.8497	0.3689	500	0.8450

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1