llama-7b_oasst1_l0.0002_64

This model is a fine-tuned version of huggyllama/llama-7b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6145

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss
1.5015	0.0018	1	1.7367
1.5123	0.3392	187	1.3207
1.1391	0.6783	374	1.3086
1.4068	1.0175	561	1.3091
1.2847	1.3566	748	1.3037
1.2433	1.6958	935	1.3003
0.9507	2.0349	1122	1.3159
1.0924	2.3741	1309	1.3710
0.9754	2.7132	1496	1.3433
0.858	3.0524	1683	1.3880
0.8205	3.3915	1870	1.3864
0.9249	3.7307	2057	1.4946
0.6185	4.0698	2244	1.5166
0.7531	4.4090	2431	1.4576
0.9268	4.7481	2618	1.4874
0.2016	5.0873	2805	1.6889
0.4437	5.4264	2992	1.6356
0.818	5.7656	3179	1.5275
0.5957	6.1047	3366	1.8285
0.2364	6.4439	3553	1.8515
0.3734	6.7830	3740	1.7053
0.3691	7.1222	3927	1.8442
0.4452	7.4613	4114	1.9495
0.2076	7.8005	4301	1.9195
0.2793	8.1397	4488	1.9103
0.2388	8.4788	4675	1.9957
0.4627	8.8180	4862	2.0253
0.1041	9.1571	5049	1.9997
0.1822	9.4963	5236	2.0561
0.242	9.8354	5423	2.1230
0.1277	10.1746	5610	2.1026
0.1238	10.5137	5797	2.1111
0.1503	10.8529	5984	2.2355
0.1341	11.1920	6171	2.2269
0.1374	11.5312	6358	2.2022
0.1162	11.8703	6545	2.3055
0.1062	12.2095	6732	2.3849
0.1457	12.5486	6919	2.2853
0.1185	12.8878	7106	2.3576
0.0897	13.2269	7293	2.4654
0.1202	13.5661	7480	2.3938
0.1729	13.9052	7667	2.3956
0.083	14.2444	7854	2.4934
0.0805	14.5835	8041	2.5021
0.1386	14.9227	8228	2.4270
0.1107	15.2618	8415	2.5474
0.0821	15.6010	8602	2.5688
0.0774	15.9401	8789	2.5323
0.0953	16.2793	8976	2.5760
0.0841	16.6185	9163	2.5870
0.0784	16.9576	9350	2.5858
0.0673	17.2968	9537	2.5586
0.131	17.6359	9724	2.5801
0.0789	17.9751	9911	2.6012

Framework versions

• PEFT 0.12.1.dev0
• Transformers 4.45.0.dev0
• Pytorch 2.3.0+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1