| 2024-04-08 20:26:22,764 INFO StreamThr :823 [internal.py:wandb_internal():86] W&B internal server running at pid: 823, started at: 2024-04-08 20:26:22.763383 | |
| 2024-04-08 20:26:22,765 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status | |
| 2024-04-08 20:26:23,135 INFO WriterThread:823 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/run-e7ipds5r.wandb | |
| 2024-04-08 20:26:23,135 DEBUG SenderThread:823 [sender.py:send():378] send: header | |
| 2024-04-08 20:26:23,138 DEBUG SenderThread:823 [sender.py:send():378] send: run | |
| 2024-04-08 20:26:23,253 INFO SenderThread:823 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files | |
| 2024-04-08 20:26:23,253 INFO SenderThread:823 [sender.py:_start_run_threads():1099] run started: e7ipds5r with start time 1712607982.763045 | |
| 2024-04-08 20:26:23,261 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-04-08 20:26:23,261 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: check_version | |
| 2024-04-08 20:26:23,350 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-04-08 20:26:23,360 DEBUG HandlerThread:823 [system_info.py:__init__():26] System info init | |
| 2024-04-08 20:26:23,360 DEBUG HandlerThread:823 [system_info.py:__init__():41] System info init done | |
| 2024-04-08 20:26:23,361 INFO HandlerThread:823 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 20:26:23,361 INFO SystemMonitor:823 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 20:26:23,361 INFO HandlerThread:823 [system_monitor.py:probe():214] Collecting system info | |
| 2024-04-08 20:26:23,361 INFO SystemMonitor:823 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 20:26:23,362 INFO SystemMonitor:823 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 20:26:23,363 INFO SystemMonitor:823 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 20:26:23,364 INFO SystemMonitor:823 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 20:26:23,365 INFO SystemMonitor:823 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-08 20:26:23,375 DEBUG HandlerThread:823 [system_info.py:probe():150] Probing system | |
| 2024-04-08 20:26:23,377 DEBUG HandlerThread:823 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-04-08 20:26:23,377 DEBUG HandlerThread:823 [system_info.py:probe():198] Probing system done | |
| 2024-04-08 20:26:23,377 DEBUG HandlerThread:823 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-08T20:26:23.375466', 'startedAt': '2024-04-08T20:26:22.756825', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '622eb14e717e', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.14, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.14, 'min': 0.0, 'max': 0.0}, {'current': 2000.14, 'min': 0.0, 'max': 0.0}, {'current': 2000.14, 'min': 0.0, 'max': 0.0}, {'current': 2000.14, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5568.727535247803}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-04-08 20:26:23,377 INFO HandlerThread:823 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-04-08 20:26:23,377 INFO HandlerThread:823 [system_monitor.py:probe():227] Publishing system info | |
| 2024-04-08 20:26:23,377 DEBUG HandlerThread:823 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-04-08 20:26:24,255 INFO Thread-12 :823 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/conda-environment.yaml | |
| 2024-04-08 20:27:18,271 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/conda-environment.yaml | |
| 2024-04-08 20:27:18,767 DEBUG HandlerThread:823 [system_info.py:_save_conda():219] Saving conda packages done | |
| 2024-04-08 20:27:18,768 INFO HandlerThread:823 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-04-08 20:27:18,773 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-04-08 20:27:18,774 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:18,775 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:18,777 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: python_packages | |
| 2024-04-08 20:27:18,780 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: stop_status | |
| 2024-04-08 20:27:18,820 DEBUG SenderThread:823 [sender.py:send():378] send: telemetry | |
| 2024-04-08 20:27:18,821 DEBUG SenderThread:823 [sender.py:send():378] send: config | |
| 2024-04-08 20:27:18,823 DEBUG SenderThread:823 [sender.py:send():378] send: metric | |
| 2024-04-08 20:27:18,823 DEBUG SenderThread:823 [sender.py:send():378] send: telemetry | |
| 2024-04-08 20:27:18,824 DEBUG SenderThread:823 [sender.py:send():378] send: metric | |
| 2024-04-08 20:27:18,824 WARNING SenderThread:823 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-04-08 20:27:18,824 DEBUG SenderThread:823 [sender.py:send():378] send: telemetry | |
| 2024-04-08 20:27:18,825 DEBUG SenderThread:823 [sender.py:send():378] send: files | |
| 2024-04-08 20:27:18,825 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-04-08 20:27:19,055 INFO wandb-upload_0:823 [upload_job.py:push():131] Uploaded file /tmp/tmpo241np2dwandb/elvgsb5j-wandb-metadata.json | |
| 2024-04-08 20:27:19,272 INFO Thread-12 :823 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/requirements.txt | |
| 2024-04-08 20:27:19,272 INFO Thread-12 :823 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-metadata.json | |
| 2024-04-08 20:27:19,272 INFO Thread-12 :823 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:27:20,499 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:21,273 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:27:23,365 DEBUG SystemMonitor:823 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 20:27:23,367 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:27:23,376 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:24,274 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/config.yaml | |
| 2024-04-08 20:27:25,510 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:28,472 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:30,182 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 20:27:30,185 DEBUG SenderThread:823 [sender.py:send():378] send: metric | |
| 2024-04-08 20:27:30,185 DEBUG SenderThread:823 [sender.py:send():378] send: metric | |
| 2024-04-08 20:27:30,185 DEBUG SenderThread:823 [sender.py:send():378] send: metric | |
| 2024-04-08 20:27:30,185 DEBUG SenderThread:823 [sender.py:send():378] send: metric | |
| 2024-04-08 20:27:30,185 DEBUG SenderThread:823 [sender.py:send():378] send: history | |
| 2024-04-08 20:27:30,185 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 20:27:30,186 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:27:30,276 INFO Thread-12 :823 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:27:30,516 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:33,278 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:27:34,037 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:35,523 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:39,038 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:40,529 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:44,039 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:45,535 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:49,040 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:50,546 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:53,368 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:27:54,374 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:27:55,286 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/config.yaml | |
| 2024-04-08 20:27:55,558 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:27:59,493 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:00,574 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:00,606 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 20:28:00,607 DEBUG SenderThread:823 [sender.py:send():378] send: history | |
| 2024-04-08 20:28:00,607 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 20:28:00,608 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:28:01,289 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:28:03,290 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:28:05,490 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:05,597 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:10,492 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:10,879 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:15,492 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:15,880 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:20,493 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:20,881 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:23,368 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:28:25,887 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:26,369 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:29,347 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 20:28:29,348 DEBUG SenderThread:823 [sender.py:send():378] send: history | |
| 2024-04-08 20:28:29,348 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 20:28:29,349 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:28:30,301 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:28:30,894 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:31,302 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:28:32,196 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:35,900 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:37,197 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:40,901 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:42,198 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:45,963 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:47,199 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:50,965 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:52,200 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:53,369 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:28:55,971 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:28:57,370 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:28:58,922 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 20:28:58,923 DEBUG SenderThread:823 [sender.py:send():378] send: history | |
| 2024-04-08 20:28:58,923 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 20:28:58,924 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:28:59,313 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:29:00,971 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:01,314 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:29:02,865 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:05,972 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:07,866 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:10,974 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:12,867 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:15,975 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:17,868 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:21,195 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:22,869 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:23,370 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:29:26,196 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:28,372 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:30,593 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 20:29:30,594 DEBUG SenderThread:823 [sender.py:send():378] send: history | |
| 2024-04-08 20:29:30,595 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 20:29:30,595 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:29:31,207 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:31,326 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:29:33,327 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:29:33,454 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:36,214 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:38,455 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:41,215 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:43,455 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:46,221 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:48,456 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:51,232 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:53,371 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:29:54,372 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:29:56,263 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:29:59,373 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:00,191 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 20:30:00,192 DEBUG SenderThread:823 [sender.py:send():378] send: history | |
| 2024-04-08 20:30:00,192 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 20:30:00,193 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:30:00,337 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:30:01,264 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:30:03,338 INFO Thread-12 :823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:30:05,118 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:06,267 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:30:10,119 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:11,268 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:30:15,120 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:16,275 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:30:20,121 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:20,722 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-08 20:30:20,723 INFO HandlerThread:823 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-08 20:30:20,723 INFO HandlerThread:823 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-08 20:30:20,723 DEBUG SystemMonitor:823 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-08 20:30:20,723 DEBUG SystemMonitor:823 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-08 20:30:20,724 INFO HandlerThread:823 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-08 20:30:20,725 INFO HandlerThread:823 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-08 20:30:20,736 INFO HandlerThread:823 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-08 20:30:20,736 INFO HandlerThread:823 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-08 20:30:20,736 INFO HandlerThread:823 [interfaces.py:finish():202] Joined network monitor | |
| 2024-04-08 20:30:20,736 DEBUG SenderThread:823 [sender.py:send():378] send: stats | |
| 2024-04-08 20:30:21,281 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 20:30:25,738 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:26,367 DEBUG SenderThread:823 [sender.py:send():378] send: exit | |
| 2024-04-08 20:30:26,367 INFO SenderThread:823 [sender.py:send_exit():585] handling exit code: 0 | |
| 2024-04-08 20:30:26,367 INFO SenderThread:823 [sender.py:send_exit():587] handling runtime: 237 | |
| 2024-04-08 20:30:26,369 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:30:26,369 INFO SenderThread:823 [sender.py:send_exit():593] send defer | |
| 2024-04-08 20:30:26,369 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,370 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 0 | |
| 2024-04-08 20:30:26,370 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,370 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 0 | |
| 2024-04-08 20:30:26,370 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 1 | |
| 2024-04-08 20:30:26,370 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,370 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 1 | |
| 2024-04-08 20:30:26,370 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,370 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 1 | |
| 2024-04-08 20:30:26,370 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 2 | |
| 2024-04-08 20:30:26,371 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,371 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 2 | |
| 2024-04-08 20:30:26,371 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,371 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 2 | |
| 2024-04-08 20:30:26,371 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 3 | |
| 2024-04-08 20:30:26,371 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,371 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 3 | |
| 2024-04-08 20:30:26,371 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,371 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 3 | |
| 2024-04-08 20:30:26,371 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 4 | |
| 2024-04-08 20:30:26,371 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,371 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 4 | |
| 2024-04-08 20:30:26,372 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,372 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 4 | |
| 2024-04-08 20:30:26,372 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 5 | |
| 2024-04-08 20:30:26,372 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,372 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 5 | |
| 2024-04-08 20:30:26,372 DEBUG SenderThread:823 [sender.py:send():378] send: summary | |
| 2024-04-08 20:30:26,373 INFO SenderThread:823 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 20:30:26,373 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,373 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 5 | |
| 2024-04-08 20:30:26,373 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 6 | |
| 2024-04-08 20:30:26,373 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,373 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 6 | |
| 2024-04-08 20:30:26,373 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,373 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 6 | |
| 2024-04-08 20:30:26,373 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 7 | |
| 2024-04-08 20:30:26,373 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 20:30:26,374 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:26,374 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 7 | |
| 2024-04-08 20:30:26,374 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:26,374 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 7 | |
| 2024-04-08 20:30:27,058 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 8 | |
| 2024-04-08 20:30:27,058 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,058 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 8 | |
| 2024-04-08 20:30:27,058 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,059 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 8 | |
| 2024-04-08 20:30:27,059 INFO SenderThread:823 [job_builder.py:build():298] Attempting to build job artifact | |
| 2024-04-08 20:30:27,060 INFO SenderThread:823 [job_builder.py:_get_source_type():439] no source found | |
| 2024-04-08 20:30:27,061 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 9 | |
| 2024-04-08 20:30:27,061 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,061 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 9 | |
| 2024-04-08 20:30:27,061 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,061 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 9 | |
| 2024-04-08 20:30:27,061 INFO SenderThread:823 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2024-04-08 20:30:27,346 INFO SenderThread:823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:30:27,347 INFO SenderThread:823 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:30:27,347 INFO SenderThread:823 [dir_watcher.py:finish():388] scan: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files | |
| 2024-04-08 20:30:27,347 INFO SenderThread:823 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/conda-environment.yaml conda-environment.yaml | |
| 2024-04-08 20:30:27,347 INFO SenderThread:823 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log output.log | |
| 2024-04-08 20:30:27,348 INFO SenderThread:823 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-metadata.json wandb-metadata.json | |
| 2024-04-08 20:30:27,348 INFO SenderThread:823 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/config.yaml config.yaml | |
| 2024-04-08 20:30:27,354 INFO SenderThread:823 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json wandb-summary.json | |
| 2024-04-08 20:30:27,354 INFO SenderThread:823 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/requirements.txt requirements.txt | |
| 2024-04-08 20:30:27,361 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 10 | |
| 2024-04-08 20:30:27,361 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,361 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 10 | |
| 2024-04-08 20:30:27,364 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,364 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 10 | |
| 2024-04-08 20:30:27,364 INFO SenderThread:823 [file_pusher.py:finish():172] shutting down file pusher | |
| 2024-04-08 20:30:27,368 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-04-08 20:30:27,369 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-04-08 20:30:27,509 INFO wandb-upload_0:823 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/output.log | |
| 2024-04-08 20:30:27,579 INFO wandb-upload_2:823 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/config.yaml | |
| 2024-04-08 20:30:27,579 INFO wandb-upload_3:823 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/wandb-summary.json | |
| 2024-04-08 20:30:27,588 INFO wandb-upload_1:823 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/conda-environment.yaml | |
| 2024-04-08 20:30:27,590 INFO wandb-upload_4:823 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240408_202622-e7ipds5r/files/requirements.txt | |
| 2024-04-08 20:30:27,790 INFO Thread-11 (_thread_body):823 [sender.py:transition_state():613] send defer: 11 | |
| 2024-04-08 20:30:27,791 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,791 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 11 | |
| 2024-04-08 20:30:27,791 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,791 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 11 | |
| 2024-04-08 20:30:27,792 INFO SenderThread:823 [file_pusher.py:join():178] waiting for file pusher | |
| 2024-04-08 20:30:27,792 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 12 | |
| 2024-04-08 20:30:27,792 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,792 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 12 | |
| 2024-04-08 20:30:27,792 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,792 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 12 | |
| 2024-04-08 20:30:27,792 INFO SenderThread:823 [file_stream.py:finish():595] file stream finish called | |
| 2024-04-08 20:30:27,839 INFO SenderThread:823 [file_stream.py:finish():599] file stream finish is done | |
| 2024-04-08 20:30:27,839 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 13 | |
| 2024-04-08 20:30:27,840 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,840 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 13 | |
| 2024-04-08 20:30:27,840 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,840 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 13 | |
| 2024-04-08 20:30:27,840 INFO SenderThread:823 [sender.py:transition_state():613] send defer: 14 | |
| 2024-04-08 20:30:27,840 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-08 20:30:27,840 INFO HandlerThread:823 [handler.py:handle_request_defer():172] handle defer: 14 | |
| 2024-04-08 20:30:27,841 DEBUG SenderThread:823 [sender.py:send():378] send: final | |
| 2024-04-08 20:30:27,841 DEBUG SenderThread:823 [sender.py:send():378] send: footer | |
| 2024-04-08 20:30:27,841 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: defer | |
| 2024-04-08 20:30:27,841 INFO SenderThread:823 [sender.py:send_request_defer():609] handle sender defer: 14 | |
| 2024-04-08 20:30:27,841 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-04-08 20:30:27,842 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-04-08 20:30:27,843 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-04-08 20:30:27,843 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: poll_exit | |
| 2024-04-08 20:30:27,844 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: server_info | |
| 2024-04-08 20:30:27,844 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: server_info | |
| 2024-04-08 20:30:27,847 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: get_summary | |
| 2024-04-08 20:30:27,847 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: sampled_history | |
| 2024-04-08 20:30:27,848 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-08 20:30:27,848 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: job_info | |
| 2024-04-08 20:30:27,862 DEBUG SenderThread:823 [sender.py:send_request():405] send_request: job_info | |
| 2024-04-08 20:30:27,862 INFO MainThread:823 [wandb_run.py:_footer_history_summary_info():3851] rendering history | |
| 2024-04-08 20:30:27,862 INFO MainThread:823 [wandb_run.py:_footer_history_summary_info():3883] rendering summary | |
| 2024-04-08 20:30:27,863 INFO MainThread:823 [wandb_run.py:_footer_sync_info():3810] logging synced files | |
| 2024-04-08 20:30:27,863 DEBUG HandlerThread:823 [handler.py:handle_request():146] handle_request: shutdown | |
| 2024-04-08 20:30:27,863 INFO HandlerThread:823 [handler.py:finish():869] shutting down handler | |
| 2024-04-08 20:30:28,849 INFO WriterThread:823 [datastore.py:close():296] close: /kaggle/working/wandb/run-20240408_202622-e7ipds5r/run-e7ipds5r.wandb | |
| 2024-04-08 20:30:28,862 INFO SenderThread:823 [sender.py:finish():1521] shutting down sender | |
| 2024-04-08 20:30:28,862 INFO SenderThread:823 [file_pusher.py:finish():172] shutting down file pusher | |
| 2024-04-08 20:30:28,862 INFO SenderThread:823 [file_pusher.py:join():178] waiting for file pusher | |