llama3-8b-instruct-qlora-medium

This model is a fine-tuned version of LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7329

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
2.2884	1.0	105	1.2658
2.0727	2.0	210	1.0205
1.9709	3.0	315	0.9518
1.8768	4.0	420	0.9206
1.7711	5.0	525	0.8761
1.6379	6.0	630	0.8487
1.4834	7.0	735	0.8200
1.3144	8.0	840	0.8076
1.1514	9.0	945	0.7972
1.0148	10.0	1050	0.7865
0.8944	11.0	1155	0.7846
0.7844	12.0	1260	0.7767
0.699	13.0	1365	0.7688
0.6215	14.0	1470	0.7631
0.5602	15.0	1575	0.7584
0.503	16.0	1680	0.7548
0.4597	17.0	1785	0.7514
0.4226	18.0	1890	0.7484
0.3903	19.0	1995	0.7441
0.3646	20.0	2100	0.7390
0.3407	21.0	2205	0.7385
0.3237	22.0	2310	0.7357
0.3108	23.0	2415	0.7343
0.2999	24.0	2520	0.7337
0.2917	25.0	2625	0.7333
0.2868	26.0	2730	0.7324
0.2815	27.0	2835	0.7327
0.28	28.0	2940	0.7315
0.2785	29.0	3045	0.7322
0.2791	30.0	3150	0.7329

Framework versions

• PEFT 0.10.0
• Transformers 4.40.0
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1