Hugging Face's logo

GaetanMichelet
/
Llama-31-8B_task-1_60-samples_config-4

PEFT
TensorBoard
Safetensors
llama
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Metrics Community
GaetanMichelet commited on
Commit
f890c0c
•
1 Parent(s): df3b474

Model save

Browse files
Files changed (5) hide show
  1. README.md +107 -0
  2. adapter_model.safetensors +1 -1
  3. all_results.json +9 -0
  4. train_results.json +9 -0
  5. trainer_state.json +829 -0
README.md ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
3
+ library_name: peft
4
+ license: llama3.1
5
+ tags:
6
+ - trl
7
+ - sft
8
+ - generated_from_trainer
9
+ model-index:
10
+ - name: Llama-31-8B_task-1_60-samples_config-4
11
+ results: []
12
+ ---
13
+
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
+
17
+ # Llama-31-8B_task-1_60-samples_config-4
18
+
19
+ This model is a fine-tuned version of [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 1.3276
22
+
23
+ ## Model description
24
+
25
+ More information needed
26
+
27
+ ## Intended uses & limitations
28
+
29
+ More information needed
30
+
31
+ ## Training and evaluation data
32
+
33
+ More information needed
34
+
35
+ ## Training procedure
36
+
37
+ ### Training hyperparameters
38
+
39
+ The following hyperparameters were used during training:
40
+ - learning_rate: 1e-05
41
+ - train_batch_size: 1
42
+ - eval_batch_size: 1
43
+ - seed: 42
44
+ - distributed_type: multi-GPU
45
+ - gradient_accumulation_steps: 16
46
+ - total_train_batch_size: 16
47
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
48
+ - lr_scheduler_type: cosine
49
+ - lr_scheduler_warmup_ratio: 0.1
50
+ - num_epochs: 150
51
+
52
+ ### Training results
53
+
54
+ | Training Loss | Epoch | Step | Validation Loss |
55
+ |:-------------:|:-------:|:----:|:---------------:|
56
+ | 2.1668 | 0.6957 | 2 | 2.0787 |
57
+ | 2.1494 | 1.7391 | 5 | 2.0737 |
58
+ | 2.127 | 2.7826 | 8 | 2.0633 |
59
+ | 2.023 | 3.8261 | 11 | 2.0470 |
60
+ | 2.0881 | 4.8696 | 14 | 2.0213 |
61
+ | 2.0378 | 5.9130 | 17 | 1.9882 |
62
+ | 2.0073 | 6.9565 | 20 | 1.9451 |
63
+ | 1.9467 | 8.0 | 23 | 1.8881 |
64
+ | 1.9133 | 8.6957 | 25 | 1.8427 |
65
+ | 1.8018 | 9.7391 | 28 | 1.7652 |
66
+ | 1.691 | 10.7826 | 31 | 1.6856 |
67
+ | 1.6135 | 11.8261 | 34 | 1.6190 |
68
+ | 1.5471 | 12.8696 | 37 | 1.5723 |
69
+ | 1.5155 | 13.9130 | 40 | 1.5440 |
70
+ | 1.4371 | 14.9565 | 43 | 1.5235 |
71
+ | 1.4825 | 16.0 | 46 | 1.5019 |
72
+ | 1.4532 | 16.6957 | 48 | 1.4894 |
73
+ | 1.4277 | 17.7391 | 51 | 1.4693 |
74
+ | 1.366 | 18.7826 | 54 | 1.4504 |
75
+ | 1.4417 | 19.8261 | 57 | 1.4330 |
76
+ | 1.3645 | 20.8696 | 60 | 1.4170 |
77
+ | 1.3153 | 21.9130 | 63 | 1.4029 |
78
+ | 1.3036 | 22.9565 | 66 | 1.3847 |
79
+ | 1.2775 | 24.0 | 69 | 1.3692 |
80
+ | 1.2726 | 24.6957 | 71 | 1.3621 |
81
+ | 1.2949 | 25.7391 | 74 | 1.3510 |
82
+ | 1.1424 | 26.7826 | 77 | 1.3406 |
83
+ | 1.2489 | 27.8261 | 80 | 1.3327 |
84
+ | 1.1662 | 28.8696 | 83 | 1.3225 |
85
+ | 1.1614 | 29.9130 | 86 | 1.3144 |
86
+ | 1.146 | 30.9565 | 89 | 1.3094 |
87
+ | 1.1177 | 32.0 | 92 | 1.3025 |
88
+ | 1.0748 | 32.6957 | 94 | 1.2985 |
89
+ | 1.118 | 33.7391 | 97 | 1.2957 |
90
+ | 1.0599 | 34.7826 | 100 | 1.2924 |
91
+ | 1.0607 | 35.8261 | 103 | 1.2912 |
92
+ | 1.0041 | 36.8696 | 106 | 1.2955 |
93
+ | 1.0132 | 37.9130 | 109 | 1.2980 |
94
+ | 1.0062 | 38.9565 | 112 | 1.3068 |
95
+ | 0.9466 | 40.0 | 115 | 1.3118 |
96
+ | 0.9728 | 40.6957 | 117 | 1.3147 |
97
+ | 0.882 | 41.7391 | 120 | 1.3195 |
98
+ | 0.9193 | 42.7826 | 123 | 1.3276 |
99
+
100
+
101
+ ### Framework versions
102
+
103
+ - PEFT 0.12.0
104
+ - Transformers 4.44.0
105
+ - Pytorch 2.1.2+cu121
106
+ - Datasets 2.20.0
107
+ - Tokenizers 0.19.1
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bea054fc3ba1300652c45fb8bf06743c41ca21ee12f0084ba4e5db09c840d0fa
3
  size 167832240
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:236562ab38ff8f7459f8eed85cb301fe359ee2721ecf2b53691dfeef1dc53291
3
  size 167832240
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 42.78260869565217,
3
+ "total_flos": 2.9607479084056576e+16,
4
+ "train_loss": 1.4191040304618152,
5
+ "train_runtime": 2316.2574,
6
+ "train_samples": 46,
7
+ "train_samples_per_second": 2.979,
8
+ "train_steps_per_second": 0.13
9
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 42.78260869565217,
3
+ "total_flos": 2.9607479084056576e+16,
4
+ "train_loss": 1.4191040304618152,
5
+ "train_runtime": 2316.2574,
6
+ "train_samples": 46,
7
+ "train_samples_per_second": 2.979,
8
+ "train_steps_per_second": 0.13
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,829 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 1.2911922931671143,
3
+ "best_model_checkpoint": "data/Llama-31-8B_task-1_60-samples_config-4/checkpoint-103",
4
+ "epoch": 42.78260869565217,
5
+ "eval_steps": 500,
6
+ "global_step": 123,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.34782608695652173,
13
+ "grad_norm": 1.8420466184616089,
14
+ "learning_rate": 3.3333333333333335e-07,
15
+ "loss": 2.0791,
16
+ "step": 1
17
+ },
18
+ {
19
+ "epoch": 0.6956521739130435,
20
+ "grad_norm": 1.7365580797195435,
21
+ "learning_rate": 6.666666666666667e-07,
22
+ "loss": 2.1668,
23
+ "step": 2
24
+ },
25
+ {
26
+ "epoch": 0.6956521739130435,
27
+ "eval_loss": 2.0786845684051514,
28
+ "eval_runtime": 4.8257,
29
+ "eval_samples_per_second": 2.487,
30
+ "eval_steps_per_second": 2.487,
31
+ "step": 2
32
+ },
33
+ {
34
+ "epoch": 1.391304347826087,
35
+ "grad_norm": 1.8509169816970825,
36
+ "learning_rate": 1.3333333333333334e-06,
37
+ "loss": 2.1494,
38
+ "step": 4
39
+ },
40
+ {
41
+ "epoch": 1.7391304347826086,
42
+ "eval_loss": 2.073704481124878,
43
+ "eval_runtime": 4.8307,
44
+ "eval_samples_per_second": 2.484,
45
+ "eval_steps_per_second": 2.484,
46
+ "step": 5
47
+ },
48
+ {
49
+ "epoch": 2.0869565217391304,
50
+ "grad_norm": 1.5950208902359009,
51
+ "learning_rate": 2.0000000000000003e-06,
52
+ "loss": 2.0264,
53
+ "step": 6
54
+ },
55
+ {
56
+ "epoch": 2.782608695652174,
57
+ "grad_norm": 1.7936512231826782,
58
+ "learning_rate": 2.666666666666667e-06,
59
+ "loss": 2.127,
60
+ "step": 8
61
+ },
62
+ {
63
+ "epoch": 2.782608695652174,
64
+ "eval_loss": 2.063272714614868,
65
+ "eval_runtime": 4.8304,
66
+ "eval_samples_per_second": 2.484,
67
+ "eval_steps_per_second": 2.484,
68
+ "step": 8
69
+ },
70
+ {
71
+ "epoch": 3.4782608695652173,
72
+ "grad_norm": 1.7478910684585571,
73
+ "learning_rate": 3.3333333333333333e-06,
74
+ "loss": 2.023,
75
+ "step": 10
76
+ },
77
+ {
78
+ "epoch": 3.8260869565217392,
79
+ "eval_loss": 2.0469565391540527,
80
+ "eval_runtime": 4.8249,
81
+ "eval_samples_per_second": 2.487,
82
+ "eval_steps_per_second": 2.487,
83
+ "step": 11
84
+ },
85
+ {
86
+ "epoch": 4.173913043478261,
87
+ "grad_norm": 1.7332440614700317,
88
+ "learning_rate": 4.000000000000001e-06,
89
+ "loss": 2.1049,
90
+ "step": 12
91
+ },
92
+ {
93
+ "epoch": 4.869565217391305,
94
+ "grad_norm": 1.879156470298767,
95
+ "learning_rate": 4.666666666666667e-06,
96
+ "loss": 2.0881,
97
+ "step": 14
98
+ },
99
+ {
100
+ "epoch": 4.869565217391305,
101
+ "eval_loss": 2.021301507949829,
102
+ "eval_runtime": 4.8272,
103
+ "eval_samples_per_second": 2.486,
104
+ "eval_steps_per_second": 2.486,
105
+ "step": 14
106
+ },
107
+ {
108
+ "epoch": 5.565217391304348,
109
+ "grad_norm": 1.6294264793395996,
110
+ "learning_rate": 5.333333333333334e-06,
111
+ "loss": 2.0378,
112
+ "step": 16
113
+ },
114
+ {
115
+ "epoch": 5.913043478260869,
116
+ "eval_loss": 1.9882330894470215,
117
+ "eval_runtime": 4.8259,
118
+ "eval_samples_per_second": 2.487,
119
+ "eval_steps_per_second": 2.487,
120
+ "step": 17
121
+ },
122
+ {
123
+ "epoch": 6.260869565217392,
124
+ "grad_norm": 1.750487208366394,
125
+ "learning_rate": 6e-06,
126
+ "loss": 1.9557,
127
+ "step": 18
128
+ },
129
+ {
130
+ "epoch": 6.956521739130435,
131
+ "grad_norm": 1.8384217023849487,
132
+ "learning_rate": 6.666666666666667e-06,
133
+ "loss": 2.0073,
134
+ "step": 20
135
+ },
136
+ {
137
+ "epoch": 6.956521739130435,
138
+ "eval_loss": 1.9450572729110718,
139
+ "eval_runtime": 4.8247,
140
+ "eval_samples_per_second": 2.487,
141
+ "eval_steps_per_second": 2.487,
142
+ "step": 20
143
+ },
144
+ {
145
+ "epoch": 7.6521739130434785,
146
+ "grad_norm": 1.9144703149795532,
147
+ "learning_rate": 7.333333333333333e-06,
148
+ "loss": 1.9467,
149
+ "step": 22
150
+ },
151
+ {
152
+ "epoch": 8.0,
153
+ "eval_loss": 1.8880866765975952,
154
+ "eval_runtime": 4.8205,
155
+ "eval_samples_per_second": 2.489,
156
+ "eval_steps_per_second": 2.489,
157
+ "step": 23
158
+ },
159
+ {
160
+ "epoch": 8.347826086956522,
161
+ "grad_norm": 1.8363193273544312,
162
+ "learning_rate": 8.000000000000001e-06,
163
+ "loss": 1.9133,
164
+ "step": 24
165
+ },
166
+ {
167
+ "epoch": 8.695652173913043,
168
+ "eval_loss": 1.8427385091781616,
169
+ "eval_runtime": 4.8315,
170
+ "eval_samples_per_second": 2.484,
171
+ "eval_steps_per_second": 2.484,
172
+ "step": 25
173
+ },
174
+ {
175
+ "epoch": 9.043478260869565,
176
+ "grad_norm": 1.9099247455596924,
177
+ "learning_rate": 8.666666666666668e-06,
178
+ "loss": 1.8557,
179
+ "step": 26
180
+ },
181
+ {
182
+ "epoch": 9.73913043478261,
183
+ "grad_norm": 1.7712645530700684,
184
+ "learning_rate": 9.333333333333334e-06,
185
+ "loss": 1.8018,
186
+ "step": 28
187
+ },
188
+ {
189
+ "epoch": 9.73913043478261,
190
+ "eval_loss": 1.765236496925354,
191
+ "eval_runtime": 4.8337,
192
+ "eval_samples_per_second": 2.483,
193
+ "eval_steps_per_second": 2.483,
194
+ "step": 28
195
+ },
196
+ {
197
+ "epoch": 10.434782608695652,
198
+ "grad_norm": 1.6165164709091187,
199
+ "learning_rate": 1e-05,
200
+ "loss": 1.691,
201
+ "step": 30
202
+ },
203
+ {
204
+ "epoch": 10.782608695652174,
205
+ "eval_loss": 1.6855782270431519,
206
+ "eval_runtime": 4.8292,
207
+ "eval_samples_per_second": 2.485,
208
+ "eval_steps_per_second": 2.485,
209
+ "step": 31
210
+ },
211
+ {
212
+ "epoch": 11.130434782608695,
213
+ "grad_norm": 1.27354097366333,
214
+ "learning_rate": 9.99864620589731e-06,
215
+ "loss": 1.738,
216
+ "step": 32
217
+ },
218
+ {
219
+ "epoch": 11.826086956521738,
220
+ "grad_norm": 1.100017786026001,
221
+ "learning_rate": 9.994585556692624e-06,
222
+ "loss": 1.6135,
223
+ "step": 34
224
+ },
225
+ {
226
+ "epoch": 11.826086956521738,
227
+ "eval_loss": 1.6189905405044556,
228
+ "eval_runtime": 4.8275,
229
+ "eval_samples_per_second": 2.486,
230
+ "eval_steps_per_second": 2.486,
231
+ "step": 34
232
+ },
233
+ {
234
+ "epoch": 12.521739130434783,
235
+ "grad_norm": 0.9115222096443176,
236
+ "learning_rate": 9.987820251299121e-06,
237
+ "loss": 1.5471,
238
+ "step": 36
239
+ },
240
+ {
241
+ "epoch": 12.869565217391305,
242
+ "eval_loss": 1.5723379850387573,
243
+ "eval_runtime": 4.8274,
244
+ "eval_samples_per_second": 2.486,
245
+ "eval_steps_per_second": 2.486,
246
+ "step": 37
247
+ },
248
+ {
249
+ "epoch": 13.217391304347826,
250
+ "grad_norm": 0.6941347122192383,
251
+ "learning_rate": 9.978353953249023e-06,
252
+ "loss": 1.5992,
253
+ "step": 38
254
+ },
255
+ {
256
+ "epoch": 13.91304347826087,
257
+ "grad_norm": 0.6947245597839355,
258
+ "learning_rate": 9.966191788709716e-06,
259
+ "loss": 1.5155,
260
+ "step": 40
261
+ },
262
+ {
263
+ "epoch": 13.91304347826087,
264
+ "eval_loss": 1.5439869165420532,
265
+ "eval_runtime": 4.8269,
266
+ "eval_samples_per_second": 2.486,
267
+ "eval_steps_per_second": 2.486,
268
+ "step": 40
269
+ },
270
+ {
271
+ "epoch": 14.608695652173914,
272
+ "grad_norm": 0.659094512462616,
273
+ "learning_rate": 9.951340343707852e-06,
274
+ "loss": 1.4371,
275
+ "step": 42
276
+ },
277
+ {
278
+ "epoch": 14.956521739130435,
279
+ "eval_loss": 1.5234659910202026,
280
+ "eval_runtime": 4.826,
281
+ "eval_samples_per_second": 2.487,
282
+ "eval_steps_per_second": 2.487,
283
+ "step": 43
284
+ },
285
+ {
286
+ "epoch": 15.304347826086957,
287
+ "grad_norm": 0.6309556365013123,
288
+ "learning_rate": 9.933807660562898e-06,
289
+ "loss": 1.4931,
290
+ "step": 44
291
+ },
292
+ {
293
+ "epoch": 16.0,
294
+ "grad_norm": 0.6867908239364624,
295
+ "learning_rate": 9.913603233532067e-06,
296
+ "loss": 1.4825,
297
+ "step": 46
298
+ },
299
+ {
300
+ "epoch": 16.0,
301
+ "eval_loss": 1.5018547773361206,
302
+ "eval_runtime": 4.825,
303
+ "eval_samples_per_second": 2.487,
304
+ "eval_steps_per_second": 2.487,
305
+ "step": 46
306
+ },
307
+ {
308
+ "epoch": 16.695652173913043,
309
+ "grad_norm": 0.6625152230262756,
310
+ "learning_rate": 9.890738003669029e-06,
311
+ "loss": 1.4532,
312
+ "step": 48
313
+ },
314
+ {
315
+ "epoch": 16.695652173913043,
316
+ "eval_loss": 1.489387035369873,
317
+ "eval_runtime": 4.8314,
318
+ "eval_samples_per_second": 2.484,
319
+ "eval_steps_per_second": 2.484,
320
+ "step": 48
321
+ },
322
+ {
323
+ "epoch": 17.391304347826086,
324
+ "grad_norm": 0.61110919713974,
325
+ "learning_rate": 9.86522435289912e-06,
326
+ "loss": 1.4277,
327
+ "step": 50
328
+ },
329
+ {
330
+ "epoch": 17.73913043478261,
331
+ "eval_loss": 1.469335913658142,
332
+ "eval_runtime": 4.8333,
333
+ "eval_samples_per_second": 2.483,
334
+ "eval_steps_per_second": 2.483,
335
+ "step": 51
336
+ },
337
+ {
338
+ "epoch": 18.08695652173913,
339
+ "grad_norm": 0.5363492369651794,
340
+ "learning_rate": 9.83707609731432e-06,
341
+ "loss": 1.4107,
342
+ "step": 52
343
+ },
344
+ {
345
+ "epoch": 18.782608695652176,
346
+ "grad_norm": 0.5772045850753784,
347
+ "learning_rate": 9.806308479691595e-06,
348
+ "loss": 1.366,
349
+ "step": 54
350
+ },
351
+ {
352
+ "epoch": 18.782608695652176,
353
+ "eval_loss": 1.45041024684906,
354
+ "eval_runtime": 4.8269,
355
+ "eval_samples_per_second": 2.486,
356
+ "eval_steps_per_second": 2.486,
357
+ "step": 54
358
+ },
359
+ {
360
+ "epoch": 19.47826086956522,
361
+ "grad_norm": 0.5792605876922607,
362
+ "learning_rate": 9.77293816123866e-06,
363
+ "loss": 1.4417,
364
+ "step": 56
365
+ },
366
+ {
367
+ "epoch": 19.82608695652174,
368
+ "eval_loss": 1.433027744293213,
369
+ "eval_runtime": 4.8255,
370
+ "eval_samples_per_second": 2.487,
371
+ "eval_steps_per_second": 2.487,
372
+ "step": 57
373
+ },
374
+ {
375
+ "epoch": 20.17391304347826,
376
+ "grad_norm": 0.5746141076087952,
377
+ "learning_rate": 9.736983212571646e-06,
378
+ "loss": 1.3183,
379
+ "step": 58
380
+ },
381
+ {
382
+ "epoch": 20.869565217391305,
383
+ "grad_norm": 0.5542544722557068,
384
+ "learning_rate": 9.698463103929542e-06,
385
+ "loss": 1.3645,
386
+ "step": 60
387
+ },
388
+ {
389
+ "epoch": 20.869565217391305,
390
+ "eval_loss": 1.4169892072677612,
391
+ "eval_runtime": 4.8302,
392
+ "eval_samples_per_second": 2.484,
393
+ "eval_steps_per_second": 2.484,
394
+ "step": 60
395
+ },
396
+ {
397
+ "epoch": 21.565217391304348,
398
+ "grad_norm": 0.5561607480049133,
399
+ "learning_rate": 9.657398694630713e-06,
400
+ "loss": 1.3153,
401
+ "step": 62
402
+ },
403
+ {
404
+ "epoch": 21.91304347826087,
405
+ "eval_loss": 1.4028769731521606,
406
+ "eval_runtime": 4.8257,
407
+ "eval_samples_per_second": 2.487,
408
+ "eval_steps_per_second": 2.487,
409
+ "step": 63
410
+ },
411
+ {
412
+ "epoch": 22.26086956521739,
413
+ "grad_norm": 0.5926358699798584,
414
+ "learning_rate": 9.613812221777212e-06,
415
+ "loss": 1.3394,
416
+ "step": 64
417
+ },
418
+ {
419
+ "epoch": 22.956521739130434,
420
+ "grad_norm": 0.5176757574081421,
421
+ "learning_rate": 9.567727288213005e-06,
422
+ "loss": 1.3036,
423
+ "step": 66
424
+ },
425
+ {
426
+ "epoch": 22.956521739130434,
427
+ "eval_loss": 1.3847342729568481,
428
+ "eval_runtime": 4.825,
429
+ "eval_samples_per_second": 2.487,
430
+ "eval_steps_per_second": 2.487,
431
+ "step": 66
432
+ },
433
+ {
434
+ "epoch": 23.652173913043477,
435
+ "grad_norm": 0.5416279435157776,
436
+ "learning_rate": 9.519168849742603e-06,
437
+ "loss": 1.2775,
438
+ "step": 68
439
+ },
440
+ {
441
+ "epoch": 24.0,
442
+ "eval_loss": 1.3691829442977905,
443
+ "eval_runtime": 4.8201,
444
+ "eval_samples_per_second": 2.49,
445
+ "eval_steps_per_second": 2.49,
446
+ "step": 69
447
+ },
448
+ {
449
+ "epoch": 24.347826086956523,
450
+ "grad_norm": 0.4943753778934479,
451
+ "learning_rate": 9.468163201617063e-06,
452
+ "loss": 1.2726,
453
+ "step": 70
454
+ },
455
+ {
456
+ "epoch": 24.695652173913043,
457
+ "eval_loss": 1.3620725870132446,
458
+ "eval_runtime": 4.8293,
459
+ "eval_samples_per_second": 2.485,
460
+ "eval_steps_per_second": 2.485,
461
+ "step": 71
462
+ },
463
+ {
464
+ "epoch": 25.043478260869566,
465
+ "grad_norm": 0.5579866766929626,
466
+ "learning_rate": 9.414737964294636e-06,
467
+ "loss": 1.2557,
468
+ "step": 72
469
+ },
470
+ {
471
+ "epoch": 25.73913043478261,
472
+ "grad_norm": 0.530767560005188,
473
+ "learning_rate": 9.358922068483813e-06,
474
+ "loss": 1.2949,
475
+ "step": 74
476
+ },
477
+ {
478
+ "epoch": 25.73913043478261,
479
+ "eval_loss": 1.3509678840637207,
480
+ "eval_runtime": 4.8304,
481
+ "eval_samples_per_second": 2.484,
482
+ "eval_steps_per_second": 2.484,
483
+ "step": 74
484
+ },
485
+ {
486
+ "epoch": 26.434782608695652,
487
+ "grad_norm": 0.5218483805656433,
488
+ "learning_rate": 9.30074573947683e-06,
489
+ "loss": 1.1424,
490
+ "step": 76
491
+ },
492
+ {
493
+ "epoch": 26.782608695652176,
494
+ "eval_loss": 1.3406327962875366,
495
+ "eval_runtime": 4.837,
496
+ "eval_samples_per_second": 2.481,
497
+ "eval_steps_per_second": 2.481,
498
+ "step": 77
499
+ },
500
+ {
501
+ "epoch": 27.130434782608695,
502
+ "grad_norm": 0.583118736743927,
503
+ "learning_rate": 9.24024048078213e-06,
504
+ "loss": 1.2227,
505
+ "step": 78
506
+ },
507
+ {
508
+ "epoch": 27.82608695652174,
509
+ "grad_norm": 0.5403833389282227,
510
+ "learning_rate": 9.177439057064684e-06,
511
+ "loss": 1.2489,
512
+ "step": 80
513
+ },
514
+ {
515
+ "epoch": 27.82608695652174,
516
+ "eval_loss": 1.3327490091323853,
517
+ "eval_runtime": 4.8298,
518
+ "eval_samples_per_second": 2.485,
519
+ "eval_steps_per_second": 2.485,
520
+ "step": 80
521
+ },
522
+ {
523
+ "epoch": 28.52173913043478,
524
+ "grad_norm": 0.5446699857711792,
525
+ "learning_rate": 9.112375476403313e-06,
526
+ "loss": 1.1662,
527
+ "step": 82
528
+ },
529
+ {
530
+ "epoch": 28.869565217391305,
531
+ "eval_loss": 1.3224714994430542,
532
+ "eval_runtime": 4.8305,
533
+ "eval_samples_per_second": 2.484,
534
+ "eval_steps_per_second": 2.484,
535
+ "step": 83
536
+ },
537
+ {
538
+ "epoch": 29.217391304347824,
539
+ "grad_norm": 0.5531767010688782,
540
+ "learning_rate": 9.045084971874738e-06,
541
+ "loss": 1.14,
542
+ "step": 84
543
+ },
544
+ {
545
+ "epoch": 29.91304347826087,
546
+ "grad_norm": 0.5570783615112305,
547
+ "learning_rate": 8.97560398247424e-06,
548
+ "loss": 1.1614,
549
+ "step": 86
550
+ },
551
+ {
552
+ "epoch": 29.91304347826087,
553
+ "eval_loss": 1.314386248588562,
554
+ "eval_runtime": 4.8261,
555
+ "eval_samples_per_second": 2.486,
556
+ "eval_steps_per_second": 2.486,
557
+ "step": 86
558
+ },
559
+ {
560
+ "epoch": 30.608695652173914,
561
+ "grad_norm": 0.5725581645965576,
562
+ "learning_rate": 8.903970133383297e-06,
563
+ "loss": 1.146,
564
+ "step": 88
565
+ },
566
+ {
567
+ "epoch": 30.956521739130434,
568
+ "eval_loss": 1.3093677759170532,
569
+ "eval_runtime": 4.8256,
570
+ "eval_samples_per_second": 2.487,
571
+ "eval_steps_per_second": 2.487,
572
+ "step": 89
573
+ },
574
+ {
575
+ "epoch": 31.304347826086957,
576
+ "grad_norm": 0.5645802617073059,
577
+ "learning_rate": 8.83022221559489e-06,
578
+ "loss": 1.1311,
579
+ "step": 90
580
+ },
581
+ {
582
+ "epoch": 32.0,
583
+ "grad_norm": 0.5483241081237793,
584
+ "learning_rate": 8.754400164907496e-06,
585
+ "loss": 1.1177,
586
+ "step": 92
587
+ },
588
+ {
589
+ "epoch": 32.0,
590
+ "eval_loss": 1.3024587631225586,
591
+ "eval_runtime": 4.8305,
592
+ "eval_samples_per_second": 2.484,
593
+ "eval_steps_per_second": 2.484,
594
+ "step": 92
595
+ },
596
+ {
597
+ "epoch": 32.69565217391305,
598
+ "grad_norm": 0.5573391914367676,
599
+ "learning_rate": 8.676545040299145e-06,
600
+ "loss": 1.0748,
601
+ "step": 94
602
+ },
603
+ {
604
+ "epoch": 32.69565217391305,
605
+ "eval_loss": 1.2985492944717407,
606
+ "eval_runtime": 4.8339,
607
+ "eval_samples_per_second": 2.482,
608
+ "eval_steps_per_second": 2.482,
609
+ "step": 94
610
+ },
611
+ {
612
+ "epoch": 33.391304347826086,
613
+ "grad_norm": 0.6121187806129456,
614
+ "learning_rate": 8.596699001693257e-06,
615
+ "loss": 1.118,
616
+ "step": 96
617
+ },
618
+ {
619
+ "epoch": 33.73913043478261,
620
+ "eval_loss": 1.2956913709640503,
621
+ "eval_runtime": 4.8239,
622
+ "eval_samples_per_second": 2.488,
623
+ "eval_steps_per_second": 2.488,
624
+ "step": 97
625
+ },
626
+ {
627
+ "epoch": 34.08695652173913,
628
+ "grad_norm": 0.6799437999725342,
629
+ "learning_rate": 8.51490528712831e-06,
630
+ "loss": 1.0753,
631
+ "step": 98
632
+ },
633
+ {
634
+ "epoch": 34.78260869565217,
635
+ "grad_norm": 0.5860521793365479,
636
+ "learning_rate": 8.43120818934367e-06,
637
+ "loss": 1.0599,
638
+ "step": 100
639
+ },
640
+ {
641
+ "epoch": 34.78260869565217,
642
+ "eval_loss": 1.29244863986969,
643
+ "eval_runtime": 4.8276,
644
+ "eval_samples_per_second": 2.486,
645
+ "eval_steps_per_second": 2.486,
646
+ "step": 100
647
+ },
648
+ {
649
+ "epoch": 35.47826086956522,
650
+ "grad_norm": 0.5923620462417603,
651
+ "learning_rate": 8.345653031794292e-06,
652
+ "loss": 1.0607,
653
+ "step": 102
654
+ },
655
+ {
656
+ "epoch": 35.82608695652174,
657
+ "eval_loss": 1.2911922931671143,
658
+ "eval_runtime": 4.8238,
659
+ "eval_samples_per_second": 2.488,
660
+ "eval_steps_per_second": 2.488,
661
+ "step": 103
662
+ },
663
+ {
664
+ "epoch": 36.17391304347826,
665
+ "grad_norm": 0.6971337199211121,
666
+ "learning_rate": 8.258286144107277e-06,
667
+ "loss": 1.0059,
668
+ "step": 104
669
+ },
670
+ {
671
+ "epoch": 36.869565217391305,
672
+ "grad_norm": 0.6799646615982056,
673
+ "learning_rate": 8.16915483699355e-06,
674
+ "loss": 1.0041,
675
+ "step": 106
676
+ },
677
+ {
678
+ "epoch": 36.869565217391305,
679
+ "eval_loss": 1.2954756021499634,
680
+ "eval_runtime": 4.8252,
681
+ "eval_samples_per_second": 2.487,
682
+ "eval_steps_per_second": 2.487,
683
+ "step": 106
684
+ },
685
+ {
686
+ "epoch": 37.56521739130435,
687
+ "grad_norm": 0.6722318530082703,
688
+ "learning_rate": 8.078307376628292e-06,
689
+ "loss": 1.0132,
690
+ "step": 108
691
+ },
692
+ {
693
+ "epoch": 37.91304347826087,
694
+ "eval_loss": 1.2979621887207031,
695
+ "eval_runtime": 4.8341,
696
+ "eval_samples_per_second": 2.482,
697
+ "eval_steps_per_second": 2.482,
698
+ "step": 109
699
+ },
700
+ {
701
+ "epoch": 38.26086956521739,
702
+ "grad_norm": 0.6754195094108582,
703
+ "learning_rate": 7.985792958513932e-06,
704
+ "loss": 0.9641,
705
+ "step": 110
706
+ },
707
+ {
708
+ "epoch": 38.95652173913044,
709
+ "grad_norm": 0.7231629490852356,
710
+ "learning_rate": 7.891661680839932e-06,
711
+ "loss": 1.0062,
712
+ "step": 112
713
+ },
714
+ {
715
+ "epoch": 38.95652173913044,
716
+ "eval_loss": 1.3067742586135864,
717
+ "eval_runtime": 4.8268,
718
+ "eval_samples_per_second": 2.486,
719
+ "eval_steps_per_second": 2.486,
720
+ "step": 112
721
+ },
722
+ {
723
+ "epoch": 39.65217391304348,
724
+ "grad_norm": 0.802635669708252,
725
+ "learning_rate": 7.795964517353734e-06,
726
+ "loss": 0.9466,
727
+ "step": 114
728
+ },
729
+ {
730
+ "epoch": 40.0,
731
+ "eval_loss": 1.3117856979370117,
732
+ "eval_runtime": 4.8215,
733
+ "eval_samples_per_second": 2.489,
734
+ "eval_steps_per_second": 2.489,
735
+ "step": 115
736
+ },
737
+ {
738
+ "epoch": 40.34782608695652,
739
+ "grad_norm": 0.8345409631729126,
740
+ "learning_rate": 7.698753289757565e-06,
741
+ "loss": 0.9728,
742
+ "step": 116
743
+ },
744
+ {
745
+ "epoch": 40.69565217391305,
746
+ "eval_loss": 1.3146535158157349,
747
+ "eval_runtime": 4.8262,
748
+ "eval_samples_per_second": 2.486,
749
+ "eval_steps_per_second": 2.486,
750
+ "step": 117
751
+ },
752
+ {
753
+ "epoch": 41.04347826086956,
754
+ "grad_norm": 0.7516780495643616,
755
+ "learning_rate": 7.600080639646077e-06,
756
+ "loss": 0.8733,
757
+ "step": 118
758
+ },
759
+ {
760
+ "epoch": 41.73913043478261,
761
+ "grad_norm": 0.778958261013031,
762
+ "learning_rate": 7.500000000000001e-06,
763
+ "loss": 0.882,
764
+ "step": 120
765
+ },
766
+ {
767
+ "epoch": 41.73913043478261,
768
+ "eval_loss": 1.319496989250183,
769
+ "eval_runtime": 4.8294,
770
+ "eval_samples_per_second": 2.485,
771
+ "eval_steps_per_second": 2.485,
772
+ "step": 120
773
+ },
774
+ {
775
+ "epoch": 42.43478260869565,
776
+ "grad_norm": 0.8342879414558411,
777
+ "learning_rate": 7.398565566251232e-06,
778
+ "loss": 0.9193,
779
+ "step": 122
780
+ },
781
+ {
782
+ "epoch": 42.78260869565217,
783
+ "eval_loss": 1.3276245594024658,
784
+ "eval_runtime": 4.8249,
785
+ "eval_samples_per_second": 2.487,
786
+ "eval_steps_per_second": 2.487,
787
+ "step": 123
788
+ },
789
+ {
790
+ "epoch": 42.78260869565217,
791
+ "step": 123,
792
+ "total_flos": 2.9607479084056576e+16,
793
+ "train_loss": 1.4191040304618152,
794
+ "train_runtime": 2316.2574,
795
+ "train_samples_per_second": 2.979,
796
+ "train_steps_per_second": 0.13
797
+ }
798
+ ],
799
+ "logging_steps": 2,
800
+ "max_steps": 300,
801
+ "num_input_tokens_seen": 0,
802
+ "num_train_epochs": 150,
803
+ "save_steps": 25,
804
+ "stateful_callbacks": {
805
+ "EarlyStoppingCallback": {
806
+ "args": {
807
+ "early_stopping_patience": 7,
808
+ "early_stopping_threshold": 0.0
809
+ },
810
+ "attributes": {
811
+ "early_stopping_patience_counter": 0
812
+ }
813
+ },
814
+ "TrainerControl": {
815
+ "args": {
816
+ "should_epoch_stop": false,
817
+ "should_evaluate": false,
818
+ "should_log": false,
819
+ "should_save": true,
820
+ "should_training_stop": true
821
+ },
822
+ "attributes": {}
823
+ }
824
+ },
825
+ "total_flos": 2.9607479084056576e+16,
826
+ "train_batch_size": 1,
827
+ "trial_name": null,
828
+ "trial_params": null
829
+ }