metadata

library_name: transformers
license: gemma
base_model: google/gemma-7b
tags:
  - trl
  - orpo
  - alignment-handbook
  - generated_from_trainer
model-index:
  - name: gemma-7b-orpo-low-quality
    results: []

gemma-7b-orpo-low-quality

This model is a fine-tuned version of google/gemma-7b on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 1.6395
Rewards/chosen: -0.0601
Rewards/rejected: -0.0755
Rewards/accuracies: 0.6029
Rewards/margins: 0.0153
Logps/rejected: -1.5091
Logps/chosen: -1.2026
Logits/rejected: 275.9735
Logits/chosen: 286.3763
Nll Loss: 1.5847
Log Odds Ratio: -0.6702
Log Odds Chosen: 0.4438

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 2
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 8
total_train_batch_size: 32
total_eval_batch_size: 2
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: inverse_sqrt
lr_scheduler_warmup_steps: 100
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen	Nll Loss	Log Odds Ratio	Log Odds Chosen
1.4933	0.9976	157	1.4686	-0.0501	-0.0608	0.5776	0.0107	-1.2166	-1.0023	307.1602	318.2524	1.4127	-0.6558	0.3240
1.036	1.9952	314	1.4194	-0.0493	-0.0612	0.5668	0.0118	-1.2231	-0.9867	302.5974	312.9305	1.3670	-0.6609	0.3487
0.56	2.9929	471	1.6395	-0.0601	-0.0755	0.6029	0.0153	-1.5091	-1.2026	275.9735	286.3763	1.5847	-0.6702	0.4438

Framework versions

Transformers 4.44.2
Pytorch 2.3.1+cu121
Datasets 2.20.0
Tokenizers 0.19.1