dpo_p69wqnv2

README.md

@@ -17,15 +17,15 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model was trained from scratch on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.7471
-- Rewards/chosen: 1.4869
-- Rewards/rejected: 1.1641
-- Rewards/accuracies: 0.4383
-- Rewards/margins: 0.3229
-- Logps/rejected: -140.3850
-- Logps/chosen: -153.8973
-- Logits/rejected: -0.6448
-- Logits/chosen: -0.6851
+- Loss: 0.6845
+- Rewards/chosen: 0.3768
+- Rewards/rejected: 0.3499
+- Rewards/accuracies: 0.5373
+- Rewards/margins: 0.0269
+- Logps/rejected: -233.9739
+- Logps/chosen: -254.7565
+- Logits/rejected: -0.6114
+- Logits/chosen: -0.7300
 
 ## Model description
 
@@ -44,7 +44,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 0.0001
+- learning_rate: 5e-06
 - train_batch_size: 1
 - eval_batch_size: 1
 - seed: 0
@@ -53,16 +53,21 @@ The following hyperparameters were used during training:
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_steps: 100
-- training_steps: 2000
+- training_steps: 2500
 
 ### Training results
 
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.6689        | 0.0567 | 750  | 0.6852          | 0.2546         | 0.2335           | 0.5373             | 0.0211          | -235.1381      | -255.9783    | -0.6108         | -0.7325       |
+| 0.6831        | 0.1133 | 1500 | 0.6835          | 0.3555         | 0.3270           | 0.5597             | 0.0285          | -234.2029      | -254.9690    | -0.6135         | -0.7317       |
+| 0.6821        | 0.1700 | 2250 | 0.6855          | 0.3655         | 0.3411           | 0.5485             | 0.0243          | -234.0616      | -254.8697    | -0.6115         | -0.7293       |
 
 
 ### Framework versions
 
 - PEFT 0.11.1
-- Transformers 4.40.2
+- Transformers 4.41.1
 - Pytorch 2.3.0+cu121
 - Datasets 2.19.1
 - Tokenizers 0.19.1

adapter_config.json

@@ -20,13 +20,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "fc_out",
+    "v_proj",
     "k_proj",
     "fc_in",
+    "wte",
     "out_proj",
-    "q_proj",
-    "v_proj",
-    "wte"
+    "fc_out",
+    "q_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d3336b30496cedb477a68def19166b41054123f3cbd0f48e23a18bc33dafcf79
+oid sha256:6827823ede9b048120358d474d275c2cce3a04d7392bb4042d9b98dfedf7e326
 size 37774720

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e8d935741839e6606b91e10fcc75f5571638a97b5871cec541b7b522e6905d6
-size 5560
+oid sha256:902272a107f94aafbb28a7b6e03093f5405a1272252662eef7575a3a9714afd7
+size 5688