Hugging Face's logo

shenxq
/
zephyr-7b-dpo-qlora-pairrm

PEFT
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
dpo
4-bit precision
bitsandbytes
Model card Files Community
shenxq commited on
Commit
c7bc043
•
1 Parent(s): 25f422b

Model save

Browse files
Files changed (6) hide show
  1. README.md +22 -26
  2. adapter_model.safetensors +1 -1
  3. all_results.json +14 -14
  4. eval_results.json +12 -12
  5. train_results.json +2 -2
  6. trainer_state.json +0 -0
README.md CHANGED
@@ -2,13 +2,9 @@
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
5
- - alignment-handbook
6
- - generated_from_trainer
7
  - trl
8
  - dpo
9
  - generated_from_trainer
10
- datasets:
11
- - snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
12
  base_model: mistralai/Mistral-7B-Instruct-v0.2
13
  model-index:
14
  - name: zephyr-7b-dpo-qlora-pairrm
@@ -20,17 +16,17 @@ should probably proofread and complete it, then remove this comment. -->
20
 
21
  # zephyr-7b-dpo-qlora-pairrm
22
 
23
- This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) on the snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset dataset.
24
  It achieves the following results on the evaluation set:
25
- - Loss: 0.6773
26
- - Rewards/chosen: -1.5442
27
- - Rewards/rejected: -1.6837
28
- - Rewards/accuracies: 0.5687
29
- - Rewards/margins: 0.1395
30
- - Logps/rejected: -394.7031
31
- - Logps/chosen: -375.1367
32
- - Logits/rejected: -4.4436
33
- - Logits/chosen: -4.4568
34
 
35
  ## Model description
36
 
@@ -65,18 +61,18 @@ The following hyperparameters were used during training:
65
 
66
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
67
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
68
- | 0.6909 | 0.08 | 100 | 0.6921 | -0.0169 | -0.0192 | 0.5427 | 0.0023 | -228.2522 | -222.4038 | -2.6219 | -2.6247 |
69
- | 0.684 | 0.16 | 200 | 0.6873 | -0.0803 | -0.0944 | 0.5567 | 0.0141 | -235.7721 | -228.7468 | -2.7599 | -2.7629 |
70
- | 0.6795 | 0.24 | 300 | 0.6839 | -0.5138 | -0.5516 | 0.5460 | 0.0378 | -281.4856 | -272.0929 | -3.5141 | -3.5199 |
71
- | 0.6561 | 0.32 | 400 | 0.6812 | -0.8158 | -0.8788 | 0.5573 | 0.0630 | -314.2105 | -302.2954 | -3.7484 | -3.7580 |
72
- | 0.633 | 0.4 | 500 | 0.6787 | -0.9027 | -0.9810 | 0.5597 | 0.0782 | -324.4269 | -310.9858 | -4.0978 | -4.1077 |
73
- | 0.6302 | 0.48 | 600 | 0.6785 | -1.1692 | -1.2692 | 0.5597 | 0.1000 | -353.2493 | -337.6355 | -4.4318 | -4.4435 |
74
- | 0.5743 | 0.56 | 700 | 0.6835 | -1.5435 | -1.6640 | 0.5630 | 0.1205 | -392.7273 | -375.0575 | -4.5047 | -4.5182 |
75
- | 0.6443 | 0.64 | 800 | 0.6779 | -1.3860 | -1.5069 | 0.5667 | 0.1209 | -377.0208 | -359.3108 | -4.2453 | -4.2572 |
76
- | 0.6651 | 0.72 | 900 | 0.6819 | -1.6633 | -1.8040 | 0.5693 | 0.1408 | -406.7332 | -387.0414 | -4.6039 | -4.6178 |
77
- | 0.5993 | 0.8 | 1000 | 0.6785 | -1.5776 | -1.7191 | 0.5683 | 0.1415 | -398.2364 | -378.4713 | -4.5356 | -4.5491 |
78
- | 0.6759 | 0.88 | 1100 | 0.6778 | -1.5515 | -1.6923 | 0.5687 | 0.1408 | -395.5604 | -375.8654 | -4.4722 | -4.4855 |
79
- | 0.6402 | 0.96 | 1200 | 0.6773 | -1.5439 | -1.6837 | 0.5690 | 0.1398 | -394.7002 | -375.1028 | -4.4444 | -4.4577 |
80
 
81
 
82
  ### Framework versions
 
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
 
 
5
  - trl
6
  - dpo
7
  - generated_from_trainer
 
 
8
  base_model: mistralai/Mistral-7B-Instruct-v0.2
9
  model-index:
10
  - name: zephyr-7b-dpo-qlora-pairrm
 
16
 
17
  # zephyr-7b-dpo-qlora-pairrm
18
 
19
+ This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) on the None dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.6749
22
+ - Rewards/chosen: -1.5494
23
+ - Rewards/rejected: -1.6972
24
+ - Rewards/accuracies: 0.5740
25
+ - Rewards/margins: 0.1478
26
+ - Logps/rejected: -396.0607
27
+ - Logps/chosen: -375.6489
28
+ - Logits/rejected: -4.3831
29
+ - Logits/chosen: -4.3963
30
 
31
  ## Model description
32
 
 
61
 
62
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
63
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
64
+ | 0.6909 | 0.08 | 100 | 0.6922 | -0.0180 | -0.0200 | 0.5237 | 0.0020 | -228.3365 | -222.5081 | -2.6230 | -2.6258 |
65
+ | 0.6843 | 0.16 | 200 | 0.6873 | -0.0792 | -0.0932 | 0.5547 | 0.0140 | -235.6526 | -228.6281 | -2.7602 | -2.7632 |
66
+ | 0.68 | 0.24 | 300 | 0.6840 | -0.5133 | -0.5505 | 0.5473 | 0.0372 | -281.3831 | -272.0374 | -3.5011 | -3.5067 |
67
+ | 0.657 | 0.32 | 400 | 0.6808 | -0.8624 | -0.9278 | 0.5583 | 0.0654 | -319.1186 | -306.9489 | -3.7427 | -3.7523 |
68
+ | 0.6342 | 0.4 | 500 | 0.6784 | -0.9237 | -1.0040 | 0.5580 | 0.0803 | -326.7317 | -313.0756 | -4.1065 | -4.1163 |
69
+ | 0.6341 | 0.48 | 600 | 0.6767 | -1.2010 | -1.3087 | 0.5630 | 0.1077 | -357.2099 | -340.8084 | -4.4445 | -4.4555 |
70
+ | 0.573 | 0.56 | 700 | 0.6808 | -1.5618 | -1.6922 | 0.5633 | 0.1304 | -395.5549 | -376.8888 | -4.4479 | -4.4607 |
71
+ | 0.6384 | 0.64 | 800 | 0.6753 | -1.3648 | -1.4912 | 0.5660 | 0.1264 | -375.4593 | -357.1873 | -4.1713 | -4.1824 |
72
+ | 0.6728 | 0.72 | 900 | 0.6791 | -1.6856 | -1.8328 | 0.5713 | 0.1471 | -409.6128 | -389.2689 | -4.5250 | -4.5392 |
73
+ | 0.603 | 0.8 | 1000 | 0.6767 | -1.6183 | -1.7689 | 0.5770 | 0.1506 | -403.2299 | -382.5415 | -4.4899 | -4.5034 |
74
+ | 0.6732 | 0.88 | 1100 | 0.6753 | -1.5670 | -1.7162 | 0.5777 | 0.1493 | -397.9568 | -377.4016 | -4.4104 | -4.4236 |
75
+ | 0.6431 | 0.96 | 1200 | 0.6750 | -1.5516 | -1.6995 | 0.5773 | 0.1479 | -396.2836 | -375.8671 | -4.3848 | -4.3979 |
76
 
77
 
78
  ### Framework versions
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5e3d24e025e43b91a16d1084ba7b450bdeeadb030ad97509691681817b168d8c
3
  size 1342238560
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e84929041b33d9d7b6b6697021d38e73783d7eeff55fb25014551bd67325dfdc
3
  size 1342238560
all_results.json CHANGED
@@ -1,20 +1,20 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": -4.456838607788086,
4
- "eval_logits/rejected": -4.443554401397705,
5
- "eval_logps/chosen": -375.13671875,
6
- "eval_logps/rejected": -394.703125,
7
- "eval_loss": 0.6773372888565063,
8
- "eval_rewards/accuracies": 0.5686666369438171,
9
- "eval_rewards/chosen": -1.5442434549331665,
10
- "eval_rewards/margins": 0.13949742913246155,
11
- "eval_rewards/rejected": -1.6837408542633057,
12
- "eval_runtime": 1623.421,
13
  "eval_samples": 2994,
14
- "eval_samples_per_second": 1.844,
15
- "eval_steps_per_second": 0.231,
16
- "train_loss": 0.6475976420174225,
17
- "train_runtime": 42677.4758,
18
  "train_samples": 19996,
19
  "train_samples_per_second": 0.469,
20
  "train_steps_per_second": 0.029
 
1
  {
2
  "epoch": 1.0,
3
+ "eval_logits/chosen": -4.396326541900635,
4
+ "eval_logits/rejected": -4.383143901824951,
5
+ "eval_logps/chosen": -375.64892578125,
6
+ "eval_logps/rejected": -396.0606689453125,
7
+ "eval_loss": 0.6749153733253479,
8
+ "eval_rewards/accuracies": 0.5740000009536743,
9
+ "eval_rewards/chosen": -1.5494239330291748,
10
+ "eval_rewards/margins": 0.14782528579235077,
11
+ "eval_rewards/rejected": -1.697249174118042,
12
+ "eval_runtime": 1692.8429,
13
  "eval_samples": 2994,
14
+ "eval_samples_per_second": 1.769,
15
+ "eval_steps_per_second": 0.222,
16
+ "train_loss": 0.647387065536218,
17
+ "train_runtime": 42673.6748,
18
  "train_samples": 19996,
19
  "train_samples_per_second": 0.469,
20
  "train_steps_per_second": 0.029
eval_results.json CHANGED
@@ -1,16 +1,16 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": -4.456838607788086,
4
- "eval_logits/rejected": -4.443554401397705,
5
- "eval_logps/chosen": -375.13671875,
6
- "eval_logps/rejected": -394.703125,
7
- "eval_loss": 0.6773372888565063,
8
- "eval_rewards/accuracies": 0.5686666369438171,
9
- "eval_rewards/chosen": -1.5442434549331665,
10
- "eval_rewards/margins": 0.13949742913246155,
11
- "eval_rewards/rejected": -1.6837408542633057,
12
- "eval_runtime": 1623.421,
13
  "eval_samples": 2994,
14
- "eval_samples_per_second": 1.844,
15
- "eval_steps_per_second": 0.231
16
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "eval_logits/chosen": -4.396326541900635,
4
+ "eval_logits/rejected": -4.383143901824951,
5
+ "eval_logps/chosen": -375.64892578125,
6
+ "eval_logps/rejected": -396.0606689453125,
7
+ "eval_loss": 0.6749153733253479,
8
+ "eval_rewards/accuracies": 0.5740000009536743,
9
+ "eval_rewards/chosen": -1.5494239330291748,
10
+ "eval_rewards/margins": 0.14782528579235077,
11
+ "eval_rewards/rejected": -1.697249174118042,
12
+ "eval_runtime": 1692.8429,
13
  "eval_samples": 2994,
14
+ "eval_samples_per_second": 1.769,
15
+ "eval_steps_per_second": 0.222
16
  }
train_results.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "epoch": 1.0,
3
- "train_loss": 0.6475976420174225,
4
- "train_runtime": 42677.4758,
5
  "train_samples": 19996,
6
  "train_samples_per_second": 0.469,
7
  "train_steps_per_second": 0.029
 
1
  {
2
  "epoch": 1.0,
3
+ "train_loss": 0.647387065536218,
4
+ "train_runtime": 42673.6748,
5
  "train_samples": 19996,
6
  "train_samples_per_second": 0.469,
7
  "train_steps_per_second": 0.029
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff