Hugging Face's logo

Femboyuwu2000
/
bloomz-1b1-vn-chat

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files Metrics Community
bloomz-1b1-vn-chat
File size: 49,262 Bytes 
291544f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16d7048
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
d809bf5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f964374
 
 
 
 
 
 
 
 
 
 
 
 
 
a8d3c55
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50d2747
 
 
 
 
 
 
 
 
 
 
 
 
c743918
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50857db
 
 
 
 
 
 
 
 
 
 
 
 
 
edd2417
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9208ac0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
357d630
 
 
 
 
 
 
 
 
 
 
 
 
70ae419
 
 
 
 
 
 
 
 
 
 
 
 
d71f063
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116b078
 
 
 
 
 
 
 
 
 
 
 
 
f7f01bd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3eba3ec
 
 
 
 
 
 
 
 
 
 
 
 
 
2671ab4
 
 
 
 
 
 
 
 
 
 
 
 
45c3630
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
c74a8fd
 
 
 
 
 
 
 
 
 
 
 
 
 
543df94
 
 
 
 
 
 
 
 
 
 
 
 
a157d5e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
 
2024-04-14 08:08:44,754 INFO    StreamThr :143 [internal.py:wandb_internal():86] W&B internal server running at pid: 143, started at: 2024-04-14 08:08:44.754030
2024-04-14 08:08:44,756 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status
2024-04-14 08:08:45,288 INFO    WriterThread:143 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/run-fw7v7n5v.wandb
2024-04-14 08:08:45,288 DEBUG   SenderThread:143 [sender.py:send():379] send: header
2024-04-14 08:08:45,297 DEBUG   SenderThread:143 [sender.py:send():379] send: run
2024-04-14 08:08:45,457 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: check_version
2024-04-14 08:08:45,459 INFO    SenderThread:143 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files
2024-04-14 08:08:45,459 INFO    SenderThread:143 [sender.py:_start_run_threads():1124] run started: fw7v7n5v with start time 1713082124.756758
2024-04-14 08:08:45,459 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: check_version
2024-04-14 08:08:45,554 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: run_start
2024-04-14 08:08:45,566 DEBUG   HandlerThread:143 [system_info.py:__init__():26] System info init
2024-04-14 08:08:45,566 DEBUG   HandlerThread:143 [system_info.py:__init__():41] System info init done
2024-04-14 08:08:45,566 INFO    HandlerThread:143 [system_monitor.py:start():194] Starting system monitor
2024-04-14 08:08:45,566 INFO    SystemMonitor:143 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-14 08:08:45,566 INFO    HandlerThread:143 [system_monitor.py:probe():214] Collecting system info
2024-04-14 08:08:45,567 INFO    SystemMonitor:143 [interfaces.py:start():190] Started cpu monitoring
2024-04-14 08:08:45,568 INFO    SystemMonitor:143 [interfaces.py:start():190] Started disk monitoring
2024-04-14 08:08:45,568 INFO    SystemMonitor:143 [interfaces.py:start():190] Started gpu monitoring
2024-04-14 08:08:45,569 INFO    SystemMonitor:143 [interfaces.py:start():190] Started memory monitoring
2024-04-14 08:08:45,570 INFO    SystemMonitor:143 [interfaces.py:start():190] Started network monitoring
2024-04-14 08:08:45,581 DEBUG   HandlerThread:143 [system_info.py:probe():150] Probing system
2024-04-14 08:08:45,583 DEBUG   HandlerThread:143 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-14 08:08:45,583 DEBUG   HandlerThread:143 [system_info.py:probe():198] Probing system done
2024-04-14 08:08:45,583 DEBUG   HandlerThread:143 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-14T08:08:45.581597', 'startedAt': '2024-04-14T08:08:44.747647', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '22f82e031f6d', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.188, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5574.93864440918}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-14 08:08:45,583 INFO    HandlerThread:143 [system_monitor.py:probe():224] Finished collecting system info
2024-04-14 08:08:45,583 INFO    HandlerThread:143 [system_monitor.py:probe():227] Publishing system info
2024-04-14 08:08:45,583 DEBUG   HandlerThread:143 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-14 08:08:46,461 INFO    Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/conda-environment.yaml
2024-04-14 08:09:00,599 ERROR   HandlerThread:143 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
    subprocess.call(
  File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
    return p.wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-14 08:09:00,605 DEBUG   HandlerThread:143 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-14 08:09:00,605 INFO    HandlerThread:143 [system_monitor.py:probe():229] Finished publishing system info
2024-04-14 08:09:00,616 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:00,616 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 08:09:00,616 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:00,616 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 08:09:00,616 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:00,616 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 08:09:00,617 DEBUG   SenderThread:143 [sender.py:send():379] send: files
2024-04-14 08:09:00,617 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-14 08:09:00,816 INFO    wandb-upload_0:143 [upload_job.py:push():131] Uploaded file /tmp/tmp6ddeqyu6wandb/52qsjx8f-wandb-metadata.json
2024-04-14 08:09:01,463 INFO    Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-metadata.json
2024-04-14 08:09:01,703 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: python_packages
2024-04-14 08:09:01,704 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: python_packages
2024-04-14 08:09:01,708 DEBUG   SenderThread:143 [sender.py:send():379] send: telemetry
2024-04-14 08:09:01,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:01,709 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:01,719 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:01,804 DEBUG   SenderThread:143 [sender.py:send():379] send: config
2024-04-14 08:09:01,807 DEBUG   SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:01,807 DEBUG   SenderThread:143 [sender.py:send():379] send: telemetry
2024-04-14 08:09:01,807 DEBUG   SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:01,807 WARNING SenderThread:143 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-14 08:09:01,807 DEBUG   SenderThread:143 [sender.py:send():379] send: telemetry
2024-04-14 08:09:02,463 INFO    Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/requirements.txt
2024-04-14 08:09:03,464 INFO    Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:09:05,465 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:09:05,923 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:10,924 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:15,929 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:16,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:16,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:16,814 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:17,470 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/config.yaml
2024-04-14 08:09:21,866 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:26,867 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:31,704 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:31,705 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:31,745 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:32,782 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:37,783 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:42,784 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:45,570 DEBUG   SystemMonitor:143 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-14 08:09:45,572 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:09:46,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:46,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:46,747 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:48,774 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:52,118 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:09:52,121 DEBUG   SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG   SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG   SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG   SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:09:52,121 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:09:52,122 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:09:52,483 INFO    Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:09:53,937 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:55,484 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:09:58,938 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:01,705 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:01,705 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:01,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:04,792 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:09,793 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:14,794 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:15,573 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:10:16,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:16,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:16,748 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:20,779 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:21,494 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/config.yaml
2024-04-14 08:10:25,885 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:30,885 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:31,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:31,708 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:31,748 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:36,803 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:41,803 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:43,062 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:10:43,063 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:10:43,063 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:10:43,065 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:10:43,502 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:10:45,503 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:10:45,574 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:10:46,705 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:46,706 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:46,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:46,840 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:51,840 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:56,841 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:01,706 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:01,706 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:01,746 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:02,799 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:03,942 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:11:03,944 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:11:03,944 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:11:03,944 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:11:04,511 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:11:07,512 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:11:08,648 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:13,649 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:15,575 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:11:16,717 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:16,717 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:16,718 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:18,816 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:23,817 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:24,756 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:11:24,757 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:11:24,757 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:11:24,759 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:11:25,520 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:11:27,521 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:11:29,691 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:31,706 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:31,706 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:31,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:34,743 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:39,744 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:44,745 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:45,576 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:11:46,706 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:46,706 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:46,747 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:49,817 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:50,426 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:11:50,427 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:11:50,427 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:11:50,429 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:11:50,531 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:11:53,533 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:11:55,104 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:00,105 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:01,706 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:01,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:01,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:05,770 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:10,772 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:12,237 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:12:12,238 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:12:12,239 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:12:12,239 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:12:12,542 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:12:15,544 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:12:15,576 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:12:16,577 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:16,706 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:16,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:16,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:21,837 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:26,838 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:31,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:31,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:31,747 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:32,804 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:37,569 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:12:37,570 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:12:37,570 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:12:37,572 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:12:38,260 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:38,553 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:12:39,554 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:12:43,262 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:45,577 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:12:46,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:46,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:46,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:48,814 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:53,814 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:58,316 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:12:58,317 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:12:58,318 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:12:58,318 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:12:58,562 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:12:58,997 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:01,564 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:13:01,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:01,707 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:01,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:04,731 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:09,731 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:14,733 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:15,578 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:13:16,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:16,708 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:16,748 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:19,828 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:24,829 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:25,132 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:13:25,134 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:13:25,134 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:13:25,136 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:13:25,574 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:13:27,575 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:13:30,810 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:31,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:31,708 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:31,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:36,736 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:41,737 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:45,579 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:13:45,910 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:13:45,911 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:13:45,912 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:13:45,912 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:13:46,582 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:13:46,707 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:46,708 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:46,708 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:46,796 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:47,582 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:13:51,797 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:56,797 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:01,708 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:01,709 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:01,749 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:02,767 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:06,631 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:14:06,632 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:14:06,633 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:14:06,633 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:14:07,589 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:14:08,286 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:09,590 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:14:13,287 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:15,580 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:14:16,708 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:16,708 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:16,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:18,798 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:23,799 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:27,910 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:14:27,913 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:14:27,913 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:14:27,914 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:14:28,597 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:14:29,587 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:29,597 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:14:31,708 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:31,709 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:31,712 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:34,830 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:39,831 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:44,832 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:45,581 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:14:46,708 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:46,709 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:46,749 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:48,932 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:14:48,933 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:14:48,934 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:14:48,934 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:14:49,605 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:14:50,677 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:51,605 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:14:55,678 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:00,679 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:01,712 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:15:01,713 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:15:01,717 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:15:05,803 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:09,725 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:15:09,726 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:15:09,726 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:15:09,727 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:15:10,612 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:15:11,438 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:11,613 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:15:15,582 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:15:16,582 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:16,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:15:16,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:15:16,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:15:21,797 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:26,798 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:30,493 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:15:30,494 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:15:30,494 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:15:30,494 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:15:30,619 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:15:31,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:15:31,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:15:31,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:15:32,758 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:33,621 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:15:37,759 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:42,760 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:45,582 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:15:46,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:15:46,709 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:15:46,749 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:15:47,836 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:51,232 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:15:51,233 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:15:51,233 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:15:51,233 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:15:51,627 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:15:52,983 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:15:53,628 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:15:57,983 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:01,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:16:01,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:16:01,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:16:03,740 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:08,741 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:11,993 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:16:11,994 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:16:11,994 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:16:11,994 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:16:12,636 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:16:13,636 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:16:14,663 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:15,588 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:16:16,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:16:16,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:16:16,713 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:16:19,844 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:24,845 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:29,846 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:31,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:16:31,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:16:31,735 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:16:32,690 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:16:32,692 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:16:32,692 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:16:32,692 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:16:33,643 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:16:35,407 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:35,644 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:16:40,408 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:45,408 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:45,584 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:16:46,709 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:16:46,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:16:46,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:16:50,833 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:16:53,409 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:16:53,410 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:16:53,410 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:16:53,412 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:16:53,652 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:16:55,653 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:16:56,109 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:01,110 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:01,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:17:01,710 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:17:01,714 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:17:06,816 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:11,818 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:14,160 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:17:14,162 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:17:14,162 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:17:14,162 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:17:14,660 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:17:15,585 DEBUG   SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:17:15,660 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:17:16,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:17:16,711 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:17:16,715 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:17:17,754 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:22,755 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:27,756 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:31,710 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:17:31,711 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:17:31,751 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:17:32,829 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:17:35,032 DEBUG   HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:17:35,033 DEBUG   SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:17:35,033 DEBUG   SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:17:35,035 INFO    SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:17:35,669 INFO    Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json