Hugging Face's logo

philschmid
/
dpo-llama-3-1-8b-math

Text Generation
Transformers
TensorBoard
Safetensors
llama
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
philschmid HF staff commited on
Commit
f39d639
·
verified ·
1 Parent(s): f7dd99d

Model save

Browse files
Files changed (11) hide show
  1. README.md +69 -0
  2. all_results.json +9 -0
  3. config.json +1 -1
  4. generation_config.json +9 -0
  5. model-00001-of-00004.safetensors +1 -1
  6. model-00002-of-00004.safetensors +1 -1
  7. model-00003-of-00004.safetensors +1 -1
  8. model-00004-of-00004.safetensors +1 -1
  9. runs/Jan23_10-35-09_ip-26-0-160-100/events.out.tfevents.1737628692.ip-26-0-160-100.1052481.0 +2 -2
  10. train_results.json +9 -0
  11. trainer_state.json +312 -0
README.md ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: philschmid/llama-3-1-8b-math-orca-spectrum-10k-ep1
3
+ library_name: transformers
4
+ model_name: dpo-llama-3-1-8b-math
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - dpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for dpo-llama-3-1-8b-math
13
+
14
+ This model is a fine-tuned version of [philschmid/llama-3-1-8b-math-orca-spectrum-10k-ep1](https://huggingface.co/philschmid/llama-3-1-8b-math-orca-spectrum-10k-ep1).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="philschmid/dpo-llama-3-1-8b-math", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+
31
+
32
+
33
+ This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
34
+
35
+ ### Framework versions
36
+
37
+ - TRL: 0.14.0.dev0
38
+ - Transformers: 4.48.1
39
+ - Pytorch: 2.5.1+cu121
40
+ - Datasets: 3.1.0
41
+ - Tokenizers: 0.21.0
42
+
43
+ ## Citations
44
+
45
+ Cite DPO as:
46
+
47
+ ```bibtex
48
+ @inproceedings{rafailov2023direct,
49
+ title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
50
+ author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
51
+ year = 2023,
52
+ booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
53
+ url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
54
+ editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
55
+ }
56
+ ```
57
+
58
+ Cite TRL as:
59
+
60
+ ```bibtex
61
+ @misc{vonwerra2022trl,
62
+ title = {{TRL: Transformer Reinforcement Learning}},
63
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
64
+ year = 2020,
65
+ journal = {GitHub repository},
66
+ publisher = {GitHub},
67
+ howpublished = {\url{https://github.com/huggingface/trl}}
68
+ }
69
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 3.6152802436582503,
5
+ "train_runtime": 1025.2608,
6
+ "train_samples": 1981,
7
+ "train_samples_per_second": 5.797,
8
+ "train_steps_per_second": 0.091
9
+ }
config.json CHANGED
@@ -31,6 +31,6 @@
31
  "tie_word_embeddings": false,
32
  "torch_dtype": "bfloat16",
33
  "transformers_version": "4.48.1",
34
- "use_cache": false,
35
  "vocab_size": 128256
36
  }
 
31
  "tie_word_embeddings": false,
32
  "torch_dtype": "bfloat16",
33
  "transformers_version": "4.48.1",
34
+ "use_cache": true,
35
  "vocab_size": 128256
36
  }
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 128000,
4
+ "do_sample": true,
5
+ "eos_token_id": 128001,
6
+ "temperature": 0.6,
7
+ "top_p": 0.9,
8
+ "transformers_version": "4.48.1"
9
+ }
model-00001-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c339411dbd38f57488392aea5be84a7dc5b6e5143620480d004b5978b83224a7
3
  size 4976698672
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd3c0dfcf504d8ee66258e702c65b779670d79d885f2c15a3604a357e9eff8c8
3
  size 4976698672
model-00002-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:edfddaf93b13f329e4dd67f095270b467408bbce00ec96cb00093f4f40b7295f
3
  size 4999802720
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:92b9f7aacd61bcb5ce1ac0bda05b76b9293c77b7c9817d207c42fe3fcac875bb
3
  size 4999802720
model-00003-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f7d57dad4a7b43125f3fbb514f866697cc640d0cbcaadb97deab93c9470e4fec
3
  size 4915916176
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62d4f655f0f58b6467d25d68b563c725c824a774b5b297fc5648d3825d771b5d
3
  size 4915916176
model-00004-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:55b3c0724711a8cc3b27fa83981fa07edb18ac063e5c5860d82a97b69df4765f
3
  size 1168138808
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e2bdaf61a638e593927939cc08178fb2cac2fa14625f539e545f11ca6c66d0e
3
  size 1168138808
runs/Jan23_10-35-09_ip-26-0-160-100/events.out.tfevents.1737628692.ip-26-0-160-100.1052481.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:25dec0fc577f41ba8b9ac1f33d5c5745d5c78f0ba1118d45482b0593af984ee9
3
- size 10406
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:20a08e384b7ca5611a2b612e38b797a2afdf97d9635f3f142007dc3da3b052ff
3
+ size 18866
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 3.6152802436582503,
5
+ "train_runtime": 1025.2608,
6
+ "train_samples": 1981,
7
+ "train_samples_per_second": 5.797,
8
+ "train_steps_per_second": 0.091
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,312 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 3.0,
5
+ "eval_steps": 500,
6
+ "global_step": 93,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.16129032258064516,
13
+ "grad_norm": 50.222183840612765,
14
+ "learning_rate": 5e-07,
15
+ "logits/chosen": -1.225976586341858,
16
+ "logits/rejected": -1.1897461414337158,
17
+ "logps/chosen": -123.3499984741211,
18
+ "logps/rejected": -135.125,
19
+ "loss": 5.5301,
20
+ "rewards/accuracies": 0.28125,
21
+ "rewards/chosen": -0.0023483275435864925,
22
+ "rewards/margins": 0.003292083740234375,
23
+ "rewards/rejected": -0.00562210101634264,
24
+ "step": 5
25
+ },
26
+ {
27
+ "epoch": 0.3225806451612903,
28
+ "grad_norm": 46.021197534461706,
29
+ "learning_rate": 5e-07,
30
+ "logits/chosen": -1.2423827648162842,
31
+ "logits/rejected": -1.1926758289337158,
32
+ "logps/chosen": -117.51249694824219,
33
+ "logps/rejected": -127.3125,
34
+ "loss": 5.5391,
35
+ "rewards/accuracies": 0.3125,
36
+ "rewards/chosen": 0.005465698428452015,
37
+ "rewards/margins": 0.0017150879139080644,
38
+ "rewards/rejected": 0.0037551880814135075,
39
+ "step": 10
40
+ },
41
+ {
42
+ "epoch": 0.4838709677419355,
43
+ "grad_norm": 46.89746866641082,
44
+ "learning_rate": 5e-07,
45
+ "logits/chosen": -1.228906273841858,
46
+ "logits/rejected": -1.205468773841858,
47
+ "logps/chosen": -121.6500015258789,
48
+ "logps/rejected": -128.0500030517578,
49
+ "loss": 5.5246,
50
+ "rewards/accuracies": 0.37187498807907104,
51
+ "rewards/chosen": 0.00469894427806139,
52
+ "rewards/margins": 0.0062500000931322575,
53
+ "rewards/rejected": -0.0015663147205486894,
54
+ "step": 15
55
+ },
56
+ {
57
+ "epoch": 0.6451612903225806,
58
+ "grad_norm": 44.09582250258885,
59
+ "learning_rate": 5e-07,
60
+ "logits/chosen": -1.206640601158142,
61
+ "logits/rejected": -1.17529296875,
62
+ "logps/chosen": -120.48750305175781,
63
+ "logps/rejected": -127.73750305175781,
64
+ "loss": 5.4723,
65
+ "rewards/accuracies": 0.44999998807907104,
66
+ "rewards/chosen": 0.02676239050924778,
67
+ "rewards/margins": 0.020679473876953125,
68
+ "rewards/rejected": 0.006100463680922985,
69
+ "step": 20
70
+ },
71
+ {
72
+ "epoch": 0.8064516129032258,
73
+ "grad_norm": 41.92350954266875,
74
+ "learning_rate": 5e-07,
75
+ "logits/chosen": -1.205664038658142,
76
+ "logits/rejected": -1.183007836341858,
77
+ "logps/chosen": -119.30000305175781,
78
+ "logps/rejected": -126.0,
79
+ "loss": 5.4928,
80
+ "rewards/accuracies": 0.4312500059604645,
81
+ "rewards/chosen": 0.03284301608800888,
82
+ "rewards/margins": 0.01783447340130806,
83
+ "rewards/rejected": 0.015004729852080345,
84
+ "step": 25
85
+ },
86
+ {
87
+ "epoch": 0.967741935483871,
88
+ "grad_norm": 42.97249437985483,
89
+ "learning_rate": 5e-07,
90
+ "logits/chosen": -1.1775391101837158,
91
+ "logits/rejected": -1.1438477039337158,
92
+ "logps/chosen": -121.01249694824219,
93
+ "logps/rejected": -130.27499389648438,
94
+ "loss": 5.4334,
95
+ "rewards/accuracies": 0.48750001192092896,
96
+ "rewards/chosen": 0.0032699585426598787,
97
+ "rewards/margins": 0.03898162767291069,
98
+ "rewards/rejected": -0.03567352145910263,
99
+ "step": 30
100
+ },
101
+ {
102
+ "epoch": 1.129032258064516,
103
+ "grad_norm": 33.66788683488558,
104
+ "learning_rate": 5e-07,
105
+ "logits/chosen": -1.1749999523162842,
106
+ "logits/rejected": -1.1409180164337158,
107
+ "logps/chosen": -118.7750015258789,
108
+ "logps/rejected": -131.21249389648438,
109
+ "loss": 4.4444,
110
+ "rewards/accuracies": 0.8356249928474426,
111
+ "rewards/chosen": 0.10396194458007812,
112
+ "rewards/margins": 0.3320721387863159,
113
+ "rewards/rejected": -0.22795777022838593,
114
+ "step": 35
115
+ },
116
+ {
117
+ "epoch": 1.2903225806451613,
118
+ "grad_norm": 30.157940166128338,
119
+ "learning_rate": 5e-07,
120
+ "logits/chosen": -1.16796875,
121
+ "logits/rejected": -1.1378905773162842,
122
+ "logps/chosen": -120.0875015258789,
123
+ "logps/rejected": -133.0124969482422,
124
+ "loss": 4.1528,
125
+ "rewards/accuracies": 0.8843749761581421,
126
+ "rewards/chosen": 0.11828994750976562,
127
+ "rewards/margins": 0.4443908631801605,
128
+ "rewards/rejected": -0.325979620218277,
129
+ "step": 40
130
+ },
131
+ {
132
+ "epoch": 1.4516129032258065,
133
+ "grad_norm": 28.845039720008064,
134
+ "learning_rate": 5e-07,
135
+ "logits/chosen": -1.1624023914337158,
136
+ "logits/rejected": -1.11767578125,
137
+ "logps/chosen": -119.375,
138
+ "logps/rejected": -133.625,
139
+ "loss": 3.8436,
140
+ "rewards/accuracies": 0.949999988079071,
141
+ "rewards/chosen": 0.09718475490808487,
142
+ "rewards/margins": 0.565844714641571,
143
+ "rewards/rejected": -0.4686523377895355,
144
+ "step": 45
145
+ },
146
+ {
147
+ "epoch": 1.6129032258064515,
148
+ "grad_norm": 29.354865646916156,
149
+ "learning_rate": 5e-07,
150
+ "logits/chosen": -1.145410180091858,
151
+ "logits/rejected": -1.113867163658142,
152
+ "logps/chosen": -119.1875,
153
+ "logps/rejected": -135.89999389648438,
154
+ "loss": 3.7449,
155
+ "rewards/accuracies": 0.921875,
156
+ "rewards/chosen": 0.025782013311982155,
157
+ "rewards/margins": 0.617114245891571,
158
+ "rewards/rejected": -0.591235339641571,
159
+ "step": 50
160
+ },
161
+ {
162
+ "epoch": 1.7741935483870968,
163
+ "grad_norm": 29.833521782828086,
164
+ "learning_rate": 5e-07,
165
+ "logits/chosen": -1.1676757335662842,
166
+ "logits/rejected": -1.1394531726837158,
167
+ "logps/chosen": -121.8375015258789,
168
+ "logps/rejected": -134.39999389648438,
169
+ "loss": 3.6219,
170
+ "rewards/accuracies": 0.9312499761581421,
171
+ "rewards/chosen": -0.0078277587890625,
172
+ "rewards/margins": 0.644726574420929,
173
+ "rewards/rejected": -0.65252685546875,
174
+ "step": 55
175
+ },
176
+ {
177
+ "epoch": 1.935483870967742,
178
+ "grad_norm": 31.512770746698088,
179
+ "learning_rate": 5e-07,
180
+ "logits/chosen": -1.18359375,
181
+ "logits/rejected": -1.137597680091858,
182
+ "logps/chosen": -121.2125015258789,
183
+ "logps/rejected": -137.0500030517578,
184
+ "loss": 3.3899,
185
+ "rewards/accuracies": 0.9468749761581421,
186
+ "rewards/chosen": -0.1258491575717926,
187
+ "rewards/margins": 0.773388683795929,
188
+ "rewards/rejected": -0.899609386920929,
189
+ "step": 60
190
+ },
191
+ {
192
+ "epoch": 2.096774193548387,
193
+ "grad_norm": 22.595598612147207,
194
+ "learning_rate": 5e-07,
195
+ "logits/chosen": -1.2158203125,
196
+ "logits/rejected": -1.17626953125,
197
+ "logps/chosen": -121.875,
198
+ "logps/rejected": -142.0124969482422,
199
+ "loss": 2.6071,
200
+ "rewards/accuracies": 0.9706250429153442,
201
+ "rewards/chosen": -0.17760467529296875,
202
+ "rewards/margins": 1.14990234375,
203
+ "rewards/rejected": -1.328222632408142,
204
+ "step": 65
205
+ },
206
+ {
207
+ "epoch": 2.258064516129032,
208
+ "grad_norm": 19.154167422104837,
209
+ "learning_rate": 5e-07,
210
+ "logits/chosen": -1.2336914539337158,
211
+ "logits/rejected": -1.1828124523162842,
212
+ "logps/chosen": -123.92500305175781,
213
+ "logps/rejected": -147.1750030517578,
214
+ "loss": 2.0145,
215
+ "rewards/accuracies": 0.9906250238418579,
216
+ "rewards/chosen": -0.23938903212547302,
217
+ "rewards/margins": 1.580664038658142,
218
+ "rewards/rejected": -1.8203125,
219
+ "step": 70
220
+ },
221
+ {
222
+ "epoch": 2.4193548387096775,
223
+ "grad_norm": 19.199349829794635,
224
+ "learning_rate": 5e-07,
225
+ "logits/chosen": -1.231835961341858,
226
+ "logits/rejected": -1.1955077648162842,
227
+ "logps/chosen": -124.19999694824219,
228
+ "logps/rejected": -151.8125,
229
+ "loss": 1.782,
230
+ "rewards/accuracies": 0.984375,
231
+ "rewards/chosen": -0.36355286836624146,
232
+ "rewards/margins": 1.8154296875,
233
+ "rewards/rejected": -2.177539110183716,
234
+ "step": 75
235
+ },
236
+ {
237
+ "epoch": 2.5806451612903225,
238
+ "grad_norm": 24.963797478719403,
239
+ "learning_rate": 5e-07,
240
+ "logits/chosen": -1.2677733898162842,
241
+ "logits/rejected": -1.2166016101837158,
242
+ "logps/chosen": -129.03750610351562,
243
+ "logps/rejected": -157.3625030517578,
244
+ "loss": 1.5875,
245
+ "rewards/accuracies": 0.984375,
246
+ "rewards/chosen": -0.652294933795929,
247
+ "rewards/margins": 2.1304688453674316,
248
+ "rewards/rejected": -2.7826170921325684,
249
+ "step": 80
250
+ },
251
+ {
252
+ "epoch": 2.741935483870968,
253
+ "grad_norm": 25.696120056907905,
254
+ "learning_rate": 5e-07,
255
+ "logits/chosen": -1.3058593273162842,
256
+ "logits/rejected": -1.253320336341858,
257
+ "logps/chosen": -130.8625030517578,
258
+ "logps/rejected": -160.22500610351562,
259
+ "loss": 1.3192,
260
+ "rewards/accuracies": 0.987500011920929,
261
+ "rewards/chosen": -0.9574218988418579,
262
+ "rewards/margins": 2.357617139816284,
263
+ "rewards/rejected": -3.315234422683716,
264
+ "step": 85
265
+ },
266
+ {
267
+ "epoch": 2.903225806451613,
268
+ "grad_norm": 21.21732885239019,
269
+ "learning_rate": 5e-07,
270
+ "logits/chosen": -1.330468773841858,
271
+ "logits/rejected": -1.279296875,
272
+ "logps/chosen": -131.71249389648438,
273
+ "logps/rejected": -169.0500030517578,
274
+ "loss": 1.1231,
275
+ "rewards/accuracies": 0.981249988079071,
276
+ "rewards/chosen": -1.255859375,
277
+ "rewards/margins": 2.912304639816284,
278
+ "rewards/rejected": -4.168359279632568,
279
+ "step": 90
280
+ },
281
+ {
282
+ "epoch": 3.0,
283
+ "step": 93,
284
+ "total_flos": 0.0,
285
+ "train_loss": 3.6152802436582503,
286
+ "train_runtime": 1025.2608,
287
+ "train_samples_per_second": 5.797,
288
+ "train_steps_per_second": 0.091
289
+ }
290
+ ],
291
+ "logging_steps": 5,
292
+ "max_steps": 93,
293
+ "num_input_tokens_seen": 0,
294
+ "num_train_epochs": 3,
295
+ "save_steps": 500,
296
+ "stateful_callbacks": {
297
+ "TrainerControl": {
298
+ "args": {
299
+ "should_epoch_stop": false,
300
+ "should_evaluate": false,
301
+ "should_log": false,
302
+ "should_save": true,
303
+ "should_training_stop": true
304
+ },
305
+ "attributes": {}
306
+ }
307
+ },
308
+ "total_flos": 0.0,
309
+ "train_batch_size": 2,
310
+ "trial_name": null,
311
+ "trial_params": null
312
+ }