openhermes-mistral-dpo-gptq

This model is a fine-tuned version of TheBloke/OpenHermes-2-Mistral-7B-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5481
• Rewards/chosen: -6.3042
• Rewards/rejected: -6.6831
• Rewards/accuracies: 0.3125
• Rewards/margins: 0.3789
• Logps/rejected: -270.7899
• Logps/chosen: -222.5794
• Logits/rejected: -2.0400
• Logits/chosen: -2.0341

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• training_steps: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6859	0.01	10	0.6666	0.0119	-0.0301	0.5	0.0419	-204.2597	-159.4191	-2.1849	-2.1781
0.683	0.01	20	0.6372	0.0384	-0.0380	0.3125	0.0763	-204.3386	-159.1542	-2.1870	-2.1765
0.7067	0.01	30	0.6194	0.0592	-0.0362	0.3125	0.0954	-204.3213	-158.9460	-2.1880	-2.1773
0.6664	0.02	40	0.6159	0.0599	-0.0264	0.3125	0.0862	-204.2225	-158.9388	-2.1856	-2.1759
0.786	0.03	50	0.5481	-6.3042	-6.6831	0.3125	0.3789	-270.7899	-222.5794	-2.0400	-2.0341

Framework versions

• Transformers 4.35.2
• Pytorch 2.0.1+cu117
• Datasets 2.15.0
• Tokenizers 0.15.0