OpenELM-1_1B-DPO-full-max-12-reward

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0566
• Rewards/chosen: -3.1406
• Rewards/rejected: -3.6719
• Rewards/accuracies: 0.5020
• Rewards/margins: 0.5234
• Logps/rejected: -656.0
• Logps/chosen: -632.0
• Logits/rejected: -16.875
• Logits/chosen: -17.0

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.0486	0.1047	100	0.7359	-1.1406	-1.5078	0.5840	0.3594	-438.0	-432.0	-12.5625	-12.75
0.0866	0.2094	200	0.7360	-1.0156	-1.1641	0.5449	0.1436	-404.0	-420.0	-12.6875	-13.0
0.0814	0.3141	300	0.7622	-1.7031	-1.8906	0.5234	0.1865	-478.0	-488.0	-6.5938	-7.2812
0.0887	0.4188	400	0.7650	-1.4688	-1.5781	0.5078	0.1094	-446.0	-466.0	-17.25	-17.125
0.0261	0.5236	500	0.8273	-1.9766	-2.1875	0.4883	0.2119	-508.0	-516.0	-17.25	-17.25
0.0476	0.6283	600	0.9206	-2.3594	-2.7031	0.5020	0.3359	-560.0	-556.0	-16.125	-16.375
0.0459	0.7330	700	0.8929	-2.0938	-2.3125	0.4707	0.2148	-520.0	-528.0	-16.5	-16.5
0.0331	0.8377	800	1.0045	-3.0938	-3.3281	0.4785	0.2207	-620.0	-628.0	-15.125	-15.4375
0.033	0.9424	900	1.0609	-3.0625	-3.3594	0.4980	0.3047	-624.0	-624.0	-13.4375	-14.0625
0.0123	1.0471	1000	1.0070	-2.9531	-3.2344	0.4922	0.2871	-612.0	-612.0	-16.375	-16.375
0.0111	1.1518	1100	0.9948	-3.0469	-3.375	0.4883	0.3340	-628.0	-624.0	-16.125	-16.25
0.001	1.2565	1200	1.0067	-2.8438	-3.2031	0.5020	0.3555	-608.0	-604.0	-16.25	-16.375
0.0078	1.3613	1300	0.9639	-2.3594	-2.6406	0.4824	0.2812	-552.0	-556.0	-15.9375	-16.125
0.0018	1.4660	1400	0.9918	-2.3281	-2.6406	0.4902	0.3027	-552.0	-552.0	-17.0	-17.0
0.0042	1.5707	1500	0.9647	-2.4375	-2.8281	0.5059	0.3926	-572.0	-564.0	-15.4375	-15.5625
0.0028	1.6754	1600	1.0578	-2.9062	-3.4375	0.5039	0.5234	-632.0	-608.0	-15.875	-16.0
0.0074	1.7801	1700	0.8793	-1.9844	-2.3281	0.5039	0.3496	-520.0	-516.0	-16.5	-16.5
0.0054	1.8848	1800	0.9229	-2.2812	-2.7031	0.5	0.4121	-560.0	-548.0	-16.875	-17.0
0.0246	1.9895	1900	1.0638	-2.9219	-3.4219	0.4902	0.5039	-632.0	-612.0	-16.75	-16.875
0.0003	2.0942	2000	1.0745	-3.0312	-3.5312	0.5020	0.4961	-644.0	-624.0	-16.75	-16.875
0.0006	2.1990	2100	1.1060	-3.3438	-3.8906	0.5059	0.5469	-676.0	-652.0	-16.625	-16.75
0.0004	2.3037	2200	1.0719	-3.2031	-3.7344	0.5059	0.5391	-664.0	-640.0	-16.75	-17.0
0.0012	2.4084	2300	1.0952	-3.3281	-3.8906	0.5020	0.5703	-680.0	-652.0	-16.75	-17.0
0.0003	2.5131	2400	1.0804	-3.2656	-3.8125	0.5	0.5547	-672.0	-644.0	-16.875	-17.0
0.0005	2.6178	2500	1.0807	-3.2656	-3.8125	0.5039	0.5547	-672.0	-644.0	-16.875	-17.0
0.0009	2.7225	2600	1.0572	-3.1406	-3.6562	0.5020	0.5234	-656.0	-632.0	-16.875	-17.0
0.0002	2.8272	2700	1.0559	-3.1406	-3.6562	0.5039	0.5273	-656.0	-632.0	-16.875	-17.0
0.0002	2.9319	2800	1.0566	-3.1406	-3.6719	0.5020	0.5234	-656.0	-632.0	-16.875	-17.0

Framework versions

• Transformers 4.45.1
• Pytorch 2.3.0
• Datasets 3.0.1
• Tokenizers 0.20.0