mmlu_math_noaugs_llama_lora

This model is a fine-tuned version of Daewon0808/prm800k_llama_fulltune on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5381
• Prm accuracy: 0.8651
• Prm precision: 0.875
• Prm recall: 0.9813
• Prm specificty: 0.2105
• Prm npv: 0.6667
• Prm f1: 0.9251
• Prm f1 neg: 0.32
• Prm f1 auc: 0.5959
• Prm f1 auc (fixed): 0.8903

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 4
• seed: 908932403
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• total_eval_batch_size: 32
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Prm accuracy	Prm precision	Prm recall	Prm specificty	Prm npv	Prm f1	Prm f1 neg	Prm f1 auc	Prm f1 auc (fixed)
No log	0	0	0.3535	0.8333	0.8772	0.9346	0.2632	0.4167	0.9050	0.3226	0.5989	0.8195
0.2836	0.0246	5	0.3544	0.8333	0.8772	0.9346	0.2632	0.4167	0.9050	0.3226	0.5989	0.8182
0.2806	0.0493	10	0.3487	0.8413	0.8718	0.9533	0.2105	0.4444	0.9107	0.2857	0.5819	0.8234
0.269	0.0739	15	0.3733	0.8571	0.8678	0.9813	0.1579	0.6	0.9211	0.25	0.5696	0.8325
0.2777	0.0985	20	0.4630	0.8571	0.856	1.0	0.0526	1.0	0.9224	0.1	0.5263	0.8524
0.211	0.1232	25	0.3822	0.8571	0.856	1.0	0.0526	1.0	0.9224	0.1	0.5263	0.8574
0.2731	0.1478	30	0.3164	0.8492	0.8729	0.9626	0.2105	0.5	0.9156	0.2963	0.5866	0.8534
0.2432	0.1724	35	0.3146	0.8492	0.8729	0.9626	0.2105	0.5	0.9156	0.2963	0.5866	0.8620
0.2814	0.1970	40	0.3221	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8706
0.2079	0.2217	45	0.2970	0.8571	0.8803	0.9626	0.2632	0.5556	0.9196	0.3571	0.6129	0.8728
0.1482	0.2463	50	0.2934	0.8651	0.8879	0.9626	0.3158	0.6	0.9238	0.4138	0.6392	0.8714
0.1697	0.2709	55	0.3032	0.8571	0.8803	0.9626	0.2632	0.5556	0.9196	0.3571	0.6129	0.8669
0.1885	0.2956	60	0.3136	0.8492	0.8667	0.9720	0.1579	0.5	0.9163	0.24	0.5649	0.8741
0.2268	0.3202	65	0.2937	0.8571	0.8870	0.9533	0.3158	0.5455	0.9189	0.4	0.6345	0.8669
0.1668	0.3448	70	0.3171	0.8492	0.8667	0.9720	0.1579	0.5	0.9163	0.24	0.5649	0.8758
0.3353	0.3695	75	0.3119	0.8492	0.8667	0.9720	0.1579	0.5	0.9163	0.24	0.5649	0.8674
0.1821	0.3941	80	0.3046	0.8413	0.8655	0.9626	0.1579	0.4286	0.9115	0.2308	0.5603	0.8699
0.2495	0.4187	85	0.3200	0.8413	0.8595	0.9720	0.1053	0.4	0.9123	0.1667	0.5386	0.8788
0.147	0.4433	90	0.2893	0.8571	0.8803	0.9626	0.2632	0.5556	0.9196	0.3571	0.6129	0.8763
0.1799	0.4680	95	0.2999	0.8651	0.8814	0.9720	0.2632	0.625	0.9244	0.3704	0.6176	0.8824
0.1852	0.4926	100	0.3146	0.8571	0.8739	0.9720	0.2105	0.5714	0.9204	0.3077	0.5912	0.8918
0.1373	0.5172	105	0.2766	0.8571	0.8803	0.9626	0.2632	0.5556	0.9196	0.3571	0.6129	0.8879
0.1716	0.5419	110	0.2689	0.8810	0.9035	0.9626	0.4211	0.6667	0.9321	0.5161	0.6918	0.8871
0.1792	0.5665	115	0.3018	0.8571	0.8739	0.9720	0.2105	0.5714	0.9204	0.3077	0.5912	0.9001
0.1899	0.5911	120	0.2957	0.8651	0.8814	0.9720	0.2632	0.625	0.9244	0.3704	0.6176	0.8930
0.2089	0.6158	125	0.2795	0.8730	0.8889	0.9720	0.3158	0.6667	0.9286	0.4286	0.6439	0.8888
0.2129	0.6404	130	0.2894	0.8651	0.8689	0.9907	0.1579	0.75	0.9258	0.2609	0.5743	0.8915
0.2025	0.6650	135	0.2856	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8886
0.2093	0.6897	140	0.2958	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8893
0.1605	0.7143	145	0.2938	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8829
0.1634	0.7389	150	0.2747	0.8651	0.8814	0.9720	0.2632	0.625	0.9244	0.3704	0.6176	0.8778
0.143	0.7635	155	0.3065	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8844
0.1322	0.7882	160	0.2988	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8842
0.2057	0.8128	165	0.2965	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8800
0.1796	0.8374	170	0.3372	0.8571	0.8618	0.9907	0.1053	0.6667	0.9217	0.1818	0.5480	0.8864
0.1777	0.8621	175	0.3096	0.8651	0.8689	0.9907	0.1579	0.75	0.9258	0.2609	0.5743	0.8829
0.1288	0.8867	180	0.2887	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8778
0.1733	0.9113	185	0.3124	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8802
0.1491	0.9360	190	0.3040	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8736
0.2341	0.9606	195	0.3299	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8731
0.159	0.9852	200	0.3395	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8751
0.0717	1.0099	205	0.3115	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8763
0.0784	1.0345	210	0.3288	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8824
0.1389	1.0591	215	0.3810	0.8571	0.8678	0.9813	0.1579	0.6	0.9211	0.25	0.5696	0.8842
0.1232	1.0837	220	0.3459	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8859
0.0996	1.1084	225	0.3710	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8871
0.0474	1.1330	230	0.3889	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8829
0.0599	1.1576	235	0.3975	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8834
0.1198	1.1823	240	0.3659	0.8730	0.8824	0.9813	0.2632	0.7143	0.9292	0.3846	0.6222	0.8842
0.1293	1.2069	245	0.3608	0.8730	0.8824	0.9813	0.2632	0.7143	0.9292	0.3846	0.6222	0.8839
0.1206	1.2315	250	0.3820	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8807
0.0965	1.2562	255	0.3273	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8795
0.1145	1.2808	260	0.3459	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8810
0.1165	1.3054	265	0.3533	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8851
0.0685	1.3300	270	0.3501	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8842
0.1442	1.3547	275	0.3470	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8898
0.1234	1.3793	280	0.3652	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8886
0.0673	1.4039	285	0.3286	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8837
0.0821	1.4286	290	0.3253	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8829
0.1246	1.4532	295	0.3151	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8864
0.0882	1.4778	300	0.3561	0.8571	0.8739	0.9720	0.2105	0.5714	0.9204	0.3077	0.5912	0.8829
0.1419	1.5025	305	0.4027	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8819
0.0634	1.5271	310	0.3743	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8871
0.1573	1.5517	315	0.3510	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8851
0.0799	1.5764	320	0.3380	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8910
0.1266	1.6010	325	0.3646	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8851
0.1259	1.6256	330	0.3471	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8874
0.0772	1.6502	335	0.3350	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8920
0.1339	1.6749	340	0.3411	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8935
0.0761	1.6995	345	0.3401	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8989
0.1278	1.7241	350	0.3467	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8982
0.0595	1.7488	355	0.3457	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8955
0.0868	1.7734	360	0.3476	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8955
0.1122	1.7980	365	0.3520	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8918
0.0634	1.8227	370	0.3592	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8930
0.0884	1.8473	375	0.3708	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8925
0.0621	1.8719	380	0.3713	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8940
0.0729	1.8966	385	0.4072	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8925
0.0641	1.9212	390	0.4109	0.8730	0.8760	0.9907	0.2105	0.8	0.9298	0.3333	0.6006	0.8876
0.0699	1.9458	395	0.3871	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8886
0.0888	1.9704	400	0.3757	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8881
0.0968	1.9951	405	0.3812	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8881
0.026	2.0197	410	0.4012	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8906
0.0429	2.0443	415	0.4218	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8938
0.0412	2.0690	420	0.4361	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8982
0.01	2.0936	425	0.4486	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8977
0.1012	2.1182	430	0.4561	0.8571	0.8739	0.9720	0.2105	0.5714	0.9204	0.3077	0.5912	0.8992
0.0518	2.1429	435	0.4691	0.8571	0.8739	0.9720	0.2105	0.5714	0.9204	0.3077	0.5912	0.9021
0.0527	2.1675	440	0.5193	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8989
0.0763	2.1921	445	0.5364	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8984
0.0222	2.2167	450	0.5032	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8967
0.0892	2.2414	455	0.4710	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8955
0.0236	2.2660	460	0.4699	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8947
0.0547	2.2906	465	0.5024	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8923
0.0177	2.3153	470	0.5419	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8915
0.0503	2.3399	475	0.5531	0.8571	0.8678	0.9813	0.1579	0.6	0.9211	0.25	0.5696	0.8923
0.0179	2.3645	480	0.5546	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8910
0.0227	2.3892	485	0.5521	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8903
0.0238	2.4138	490	0.5510	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8925
0.0448	2.4384	495	0.5606	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8908
0.0558	2.4631	500	0.5546	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8925
0.0647	2.4877	505	0.5585	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8910
0.0238	2.5123	510	0.5570	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8891
0.0574	2.5369	515	0.5500	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8898
0.0407	2.5616	520	0.5473	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8901
0.021	2.5862	525	0.5454	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8901
0.0203	2.6108	530	0.5482	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8881
0.0049	2.6355	535	0.5464	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8901
0.0483	2.6601	540	0.5463	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8893
0.02	2.6847	545	0.5462	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8881
0.0166	2.7094	550	0.5434	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8886
0.0283	2.7340	555	0.5440	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8906
0.0056	2.7586	560	0.5415	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8883
0.0246	2.7833	565	0.5409	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8901
0.054	2.8079	570	0.5360	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8879
0.0309	2.8325	575	0.5436	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8891
0.0276	2.8571	580	0.5374	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8893
0.0344	2.8818	585	0.5395	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8886
0.0116	2.9064	590	0.5385	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8896
0.0169	2.9310	595	0.5359	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8908
0.0115	2.9557	600	0.5375	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8869
0.0563	2.9803	605	0.5381	0.8651	0.875	0.9813	0.2105	0.6667	0.9251	0.32	0.5959	0.8903

Framework versions

• PEFT 0.12.0
• Transformers 4.46.0
• Pytorch 2.4.0+cu118
• Datasets 3.0.0
• Tokenizers 0.20.1