| 2024-04-09 21:46:03,067 INFO StreamThr :160 [internal.py:wandb_internal():86] W&B internal server running at pid: 160, started at: 2024-04-09 21:46:03.066337 | |
| 2024-04-09 21:46:03,068 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status | |
| 2024-04-09 21:46:03,465 INFO WriterThread:160 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/run-e7cki9vp.wandb | |
| 2024-04-09 21:46:03,466 DEBUG SenderThread:160 [sender.py:send():379] send: header | |
| 2024-04-09 21:46:03,470 DEBUG SenderThread:160 [sender.py:send():379] send: run | |
| 2024-04-09 21:46:03,649 INFO SenderThread:160 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files | |
| 2024-04-09 21:46:03,649 INFO SenderThread:160 [sender.py:_start_run_threads():1124] run started: e7cki9vp with start time 1712699163.06746 | |
| 2024-04-09 21:46:03,658 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-04-09 21:46:03,658 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: check_version | |
| 2024-04-09 21:46:03,756 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-04-09 21:46:03,768 DEBUG HandlerThread:160 [system_info.py:__init__():26] System info init | |
| 2024-04-09 21:46:03,768 DEBUG HandlerThread:160 [system_info.py:__init__():41] System info init done | |
| 2024-04-09 21:46:03,768 INFO HandlerThread:160 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-09 21:46:03,768 INFO SystemMonitor:160 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-09 21:46:03,768 INFO HandlerThread:160 [system_monitor.py:probe():214] Collecting system info | |
| 2024-04-09 21:46:03,769 INFO SystemMonitor:160 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-09 21:46:03,769 INFO SystemMonitor:160 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-09 21:46:03,770 INFO SystemMonitor:160 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-09 21:46:03,771 INFO SystemMonitor:160 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-09 21:46:03,771 INFO SystemMonitor:160 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-09 21:46:03,782 DEBUG HandlerThread:160 [system_info.py:probe():150] Probing system | |
| 2024-04-09 21:46:03,785 DEBUG HandlerThread:160 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-04-09 21:46:03,785 DEBUG HandlerThread:160 [system_info.py:probe():198] Probing system done | |
| 2024-04-09 21:46:03,785 DEBUG HandlerThread:160 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-09T21:46:03.782958', 'startedAt': '2024-04-09T21:46:03.059424', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '6e44b39f6877', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.152, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.152, 'min': 0.0, 'max': 0.0}, {'current': 2000.152, 'min': 0.0, 'max': 0.0}, {'current': 2000.152, 'min': 0.0, 'max': 0.0}, {'current': 2000.152, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5568.826061248779}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-04-09 21:46:03,785 INFO HandlerThread:160 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-04-09 21:46:03,785 INFO HandlerThread:160 [system_monitor.py:probe():227] Publishing system info | |
| 2024-04-09 21:46:03,785 DEBUG HandlerThread:160 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-04-09 21:46:04,652 INFO Thread-12 :160 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/conda-environment.yaml | |
| 2024-04-09 21:46:18,800 ERROR HandlerThread:160 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds | |
| Traceback (most recent call last): | |
| File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda | |
| subprocess.call( | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call | |
| return p.wait(timeout=timeout) | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait | |
| return self._wait(timeout=timeout) | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait | |
| raise TimeoutExpired(self.args, timeout) | |
| subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds | |
| 2024-04-09 21:46:18,802 DEBUG HandlerThread:160 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-04-09 21:46:18,803 INFO HandlerThread:160 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-04-09 21:46:18,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:18,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-09 21:46:18,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:18,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-09 21:46:18,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:18,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-09 21:46:18,811 DEBUG SenderThread:160 [sender.py:send():379] send: files | |
| 2024-04-09 21:46:18,811 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now | |
| 2024-04-09 21:46:19,179 INFO wandb-upload_0:160 [upload_job.py:push():131] Uploaded file /tmp/tmp92pjcj01wandb/kvu2hqxk-wandb-metadata.json | |
| 2024-04-09 21:46:19,654 INFO Thread-12 :160 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-metadata.json | |
| 2024-04-09 21:46:19,806 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-04-09 21:46:19,806 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: python_packages | |
| 2024-04-09 21:46:19,810 DEBUG SenderThread:160 [sender.py:send():379] send: telemetry | |
| 2024-04-09 21:46:19,812 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:46:19,812 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:46:19,862 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:46:19,872 DEBUG SenderThread:160 [sender.py:send():379] send: config | |
| 2024-04-09 21:46:19,874 DEBUG SenderThread:160 [sender.py:send():379] send: metric | |
| 2024-04-09 21:46:19,874 DEBUG SenderThread:160 [sender.py:send():379] send: telemetry | |
| 2024-04-09 21:46:19,874 DEBUG SenderThread:160 [sender.py:send():379] send: metric | |
| 2024-04-09 21:46:19,874 WARNING SenderThread:160 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) | |
| 2024-04-09 21:46:19,874 DEBUG SenderThread:160 [sender.py:send():379] send: telemetry | |
| 2024-04-09 21:46:20,655 INFO Thread-12 :160 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/requirements.txt | |
| 2024-04-09 21:46:20,655 INFO Thread-12 :160 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:46:22,655 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:46:24,034 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:29,035 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:34,041 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:34,660 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/config.yaml | |
| 2024-04-09 21:46:34,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:46:34,810 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:46:34,811 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:46:39,961 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:44,962 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:49,807 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:46:49,807 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:46:49,847 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:46:50,920 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:46:55,921 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:00,922 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:03,772 DEBUG SystemMonitor:160 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-09 21:47:03,774 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:47:04,807 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:47:04,807 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:47:04,848 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:47:05,925 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:10,926 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:14,340 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:47:14,342 DEBUG SenderThread:160 [sender.py:send():379] send: metric | |
| 2024-04-09 21:47:14,342 DEBUG SenderThread:160 [sender.py:send():379] send: metric | |
| 2024-04-09 21:47:14,343 DEBUG SenderThread:160 [sender.py:send():379] send: metric | |
| 2024-04-09 21:47:14,343 DEBUG SenderThread:160 [sender.py:send():379] send: metric | |
| 2024-04-09 21:47:14,343 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:47:14,343 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:47:14,345 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:47:14,675 INFO Thread-12 :160 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:47:16,676 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:47:16,676 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:19,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:47:19,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:47:19,809 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:47:21,900 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:26,901 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:31,902 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:33,775 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:47:34,807 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:47:34,807 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:47:34,848 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:47:37,879 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:38,684 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/config.yaml | |
| 2024-04-09 21:47:42,992 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:47,992 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:49,807 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:47:49,808 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:47:49,848 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:47:53,904 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:47:58,905 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:03,776 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:48:04,777 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:04,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:48:04,808 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:48:04,849 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:48:09,962 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:12,408 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:48:12,409 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:48:12,409 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:48:12,409 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:48:12,699 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:48:14,700 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:48:15,653 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:19,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:48:19,808 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:48:19,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:48:20,941 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:25,943 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:30,943 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:33,777 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:48:34,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:48:34,808 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:48:34,849 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:48:35,949 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:40,950 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:45,951 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:49,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:48:49,808 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:48:49,849 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:48:50,961 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:48:55,962 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:00,963 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:03,779 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:49:04,458 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:49:04,459 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:49:04,459 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:49:04,460 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:49:04,719 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:49:04,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:49:04,809 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:49:04,812 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:49:06,719 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:49:06,903 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:11,904 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:16,904 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:19,808 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:49:19,809 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:49:19,849 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:49:21,960 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:26,961 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:31,962 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:33,779 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:49:34,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:49:34,809 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:49:34,849 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:49:37,903 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:42,904 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:47,905 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:49,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:49:49,809 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:49:49,850 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:49:52,947 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:49:57,948 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:02,949 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:03,253 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:50:03,254 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:50:03,254 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:50:03,256 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:50:03,742 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:50:03,781 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:50:04,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:50:04,810 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:50:04,813 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:50:06,743 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:50:08,908 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:13,909 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:18,910 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:19,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:50:19,810 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:50:19,850 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:50:23,921 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:28,922 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:33,782 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:50:34,783 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:34,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:50:34,809 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:50:34,850 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:50:39,971 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:44,972 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:49,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:50:49,810 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:50:49,850 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:50:50,889 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:50:55,890 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:00,589 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:51:00,590 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:51:00,590 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:51:00,592 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:51:00,764 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:51:00,908 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:02,765 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:51:03,783 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:51:04,809 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:51:04,810 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:51:04,813 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:51:05,944 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:10,945 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:15,946 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:19,830 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:51:19,841 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:51:19,842 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:51:20,989 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:25,990 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:30,991 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:33,783 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:51:34,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:51:34,816 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:51:34,819 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:51:36,912 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:41,914 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:46,913 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:49,815 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:51:49,816 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:51:49,856 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:51:51,966 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:55,844 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:51:55,845 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:51:55,845 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:51:55,845 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:51:56,784 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:51:57,094 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:51:58,785 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:52:02,095 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:03,784 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:52:04,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:52:04,816 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:52:04,820 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:52:07,900 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:12,900 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:17,901 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:19,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:52:19,816 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:52:19,857 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:52:23,893 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:28,893 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:33,785 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:52:34,786 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:34,818 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:52:34,818 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:52:34,818 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:52:39,947 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:44,947 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:49,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:52:49,817 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:52:49,817 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:52:50,486 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:52:50,487 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:52:50,487 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:52:50,487 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:52:50,488 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:52:50,805 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:52:52,806 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:52:55,739 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:00,740 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:03,786 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:53:04,817 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:53:04,817 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:53:04,820 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:53:05,903 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:10,904 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:15,905 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:19,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:53:19,816 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:53:19,857 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:53:21,885 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:26,886 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:31,887 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:33,787 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:53:34,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:53:34,816 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:53:34,857 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:53:36,987 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:41,988 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:43,389 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:53:43,390 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:53:43,390 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:53:43,391 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:53:43,829 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:53:44,829 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:53:47,638 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:49,816 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:53:49,817 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:53:49,820 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:53:52,887 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:53:57,888 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:02,889 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:03,788 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:54:04,844 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:54:04,845 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:54:04,846 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:54:07,926 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:12,927 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:17,928 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:19,824 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:54:19,824 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:54:19,828 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:54:23,918 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:28,919 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:33,789 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:54:34,790 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:34,824 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:54:34,825 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:54:34,865 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:54:36,955 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-09 21:54:36,956 DEBUG SenderThread:160 [sender.py:send():379] send: history | |
| 2024-04-09 21:54:36,957 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-09 21:54:36,957 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:54:37,849 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:54:38,850 INFO Thread-12 :160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:54:40,220 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:45,221 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:49,822 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-09 21:54:49,823 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-09 21:54:49,826 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:54:50,962 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:55,963 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:54:58,795 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-09 21:54:58,796 INFO HandlerThread:160 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-09 21:54:58,796 INFO HandlerThread:160 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-09 21:54:58,796 DEBUG SystemMonitor:160 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-09 21:54:58,796 INFO HandlerThread:160 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-09 21:54:58,797 DEBUG SystemMonitor:160 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-09 21:54:58,797 INFO HandlerThread:160 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-09 21:54:58,807 INFO HandlerThread:160 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-09 21:54:58,808 INFO HandlerThread:160 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-09 21:54:58,808 INFO HandlerThread:160 [interfaces.py:finish():202] Joined network monitor | |
| 2024-04-09 21:54:58,812 DEBUG SenderThread:160 [sender.py:send():379] send: stats | |
| 2024-04-09 21:55:01,813 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:55:02,316 DEBUG SenderThread:160 [sender.py:send():379] send: exit | |
| 2024-04-09 21:55:02,316 INFO SenderThread:160 [sender.py:send_exit():586] handling exit code: 0 | |
| 2024-04-09 21:55:02,316 INFO SenderThread:160 [sender.py:send_exit():588] handling runtime: 535 | |
| 2024-04-09 21:55:02,317 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:55:02,317 INFO SenderThread:160 [sender.py:send_exit():594] send defer | |
| 2024-04-09 21:55:02,317 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,317 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 0 | |
| 2024-04-09 21:55:02,318 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,318 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 0 | |
| 2024-04-09 21:55:02,318 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 1 | |
| 2024-04-09 21:55:02,318 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,318 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 1 | |
| 2024-04-09 21:55:02,318 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,318 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 1 | |
| 2024-04-09 21:55:02,318 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 2 | |
| 2024-04-09 21:55:02,318 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,318 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 2 | |
| 2024-04-09 21:55:02,318 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,318 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 2 | |
| 2024-04-09 21:55:02,319 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 3 | |
| 2024-04-09 21:55:02,319 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,319 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 3 | |
| 2024-04-09 21:55:02,319 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,319 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 3 | |
| 2024-04-09 21:55:02,319 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 4 | |
| 2024-04-09 21:55:02,319 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,319 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 4 | |
| 2024-04-09 21:55:02,319 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,319 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 4 | |
| 2024-04-09 21:55:02,319 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 5 | |
| 2024-04-09 21:55:02,319 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,320 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 5 | |
| 2024-04-09 21:55:02,320 DEBUG SenderThread:160 [sender.py:send():379] send: summary | |
| 2024-04-09 21:55:02,320 INFO SenderThread:160 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-09 21:55:02,321 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,321 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 5 | |
| 2024-04-09 21:55:02,321 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 6 | |
| 2024-04-09 21:55:02,321 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,321 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 6 | |
| 2024-04-09 21:55:02,321 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,321 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 6 | |
| 2024-04-09 21:55:02,321 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 7 | |
| 2024-04-09 21:55:02,321 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-09 21:55:02,321 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,321 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 7 | |
| 2024-04-09 21:55:02,322 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,322 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 7 | |
| 2024-04-09 21:55:02,501 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 8 | |
| 2024-04-09 21:55:02,501 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,501 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 8 | |
| 2024-04-09 21:55:02,502 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,502 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 8 | |
| 2024-04-09 21:55:02,502 INFO SenderThread:160 [job_builder.py:build():318] Attempting to build job artifact | |
| 2024-04-09 21:55:02,504 INFO SenderThread:160 [job_builder.py:_get_source_type():466] no source found | |
| 2024-04-09 21:55:02,504 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 9 | |
| 2024-04-09 21:55:02,504 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,504 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 9 | |
| 2024-04-09 21:55:02,504 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,504 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 9 | |
| 2024-04-09 21:55:02,504 INFO SenderThread:160 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2024-04-09 21:55:02,855 INFO SenderThread:160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:55:02,855 INFO SenderThread:160 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:55:02,855 INFO SenderThread:160 [dir_watcher.py:finish():388] scan: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files | |
| 2024-04-09 21:55:02,856 INFO SenderThread:160 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/conda-environment.yaml conda-environment.yaml | |
| 2024-04-09 21:55:02,856 INFO SenderThread:160 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/requirements.txt requirements.txt | |
| 2024-04-09 21:55:02,856 INFO SenderThread:160 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-metadata.json wandb-metadata.json | |
| 2024-04-09 21:55:02,856 INFO SenderThread:160 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json wandb-summary.json | |
| 2024-04-09 21:55:02,856 INFO SenderThread:160 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/config.yaml config.yaml | |
| 2024-04-09 21:55:02,862 INFO SenderThread:160 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log output.log | |
| 2024-04-09 21:55:02,867 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 10 | |
| 2024-04-09 21:55:02,870 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:02,870 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 10 | |
| 2024-04-09 21:55:02,871 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:02,871 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 10 | |
| 2024-04-09 21:55:02,871 INFO SenderThread:160 [file_pusher.py:finish():172] shutting down file pusher | |
| 2024-04-09 21:55:03,144 INFO wandb-upload_0:160 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/requirements.txt | |
| 2024-04-09 21:55:03,156 INFO wandb-upload_3:160 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/output.log | |
| 2024-04-09 21:55:03,171 INFO wandb-upload_1:160 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/wandb-summary.json | |
| 2024-04-09 21:55:03,185 INFO wandb-upload_2:160 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240409_214603-e7cki9vp/files/config.yaml | |
| 2024-04-09 21:55:03,316 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-04-09 21:55:03,317 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: poll_exit | |
| 2024-04-09 21:55:03,385 INFO Thread-11 (_thread_body):160 [sender.py:transition_state():614] send defer: 11 | |
| 2024-04-09 21:55:03,386 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:03,386 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 11 | |
| 2024-04-09 21:55:03,386 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:03,386 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 11 | |
| 2024-04-09 21:55:03,387 INFO SenderThread:160 [file_pusher.py:join():178] waiting for file pusher | |
| 2024-04-09 21:55:03,387 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 12 | |
| 2024-04-09 21:55:03,387 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:03,387 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 12 | |
| 2024-04-09 21:55:03,387 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:03,387 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 12 | |
| 2024-04-09 21:55:03,387 INFO SenderThread:160 [file_stream.py:finish():614] file stream finish called | |
| 2024-04-09 21:55:03,572 INFO SenderThread:160 [file_stream.py:finish():618] file stream finish is done | |
| 2024-04-09 21:55:03,572 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 13 | |
| 2024-04-09 21:55:03,572 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:03,572 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 13 | |
| 2024-04-09 21:55:03,573 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:03,573 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 13 | |
| 2024-04-09 21:55:03,573 INFO SenderThread:160 [sender.py:transition_state():614] send defer: 14 | |
| 2024-04-09 21:55:03,573 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: defer | |
| 2024-04-09 21:55:03,573 INFO HandlerThread:160 [handler.py:handle_request_defer():172] handle defer: 14 | |
| 2024-04-09 21:55:03,573 DEBUG SenderThread:160 [sender.py:send():379] send: final | |
| 2024-04-09 21:55:03,574 DEBUG SenderThread:160 [sender.py:send():379] send: footer | |
| 2024-04-09 21:55:03,574 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: defer | |
| 2024-04-09 21:55:03,574 INFO SenderThread:160 [sender.py:send_request_defer():610] handle sender defer: 14 | |
| 2024-04-09 21:55:03,575 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-04-09 21:55:03,575 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: poll_exit | |
| 2024-04-09 21:55:03,576 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2024-04-09 21:55:03,576 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: poll_exit | |
| 2024-04-09 21:55:03,577 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: server_info | |
| 2024-04-09 21:55:03,577 DEBUG SenderThread:160 [sender.py:send_request():406] send_request: server_info | |
| 2024-04-09 21:55:03,580 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: get_summary | |
| 2024-04-09 21:55:03,580 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: sampled_history | |
| 2024-04-09 21:55:03,581 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-09 21:55:03,648 INFO MainThread:160 [wandb_run.py:_footer_history_summary_info():3920] rendering history | |
| 2024-04-09 21:55:03,648 INFO MainThread:160 [wandb_run.py:_footer_history_summary_info():3952] rendering summary | |
| 2024-04-09 21:55:03,648 INFO MainThread:160 [wandb_run.py:_footer_sync_info():3879] logging synced files | |
| 2024-04-09 21:55:03,649 DEBUG HandlerThread:160 [handler.py:handle_request():146] handle_request: shutdown | |
| 2024-04-09 21:55:03,649 INFO HandlerThread:160 [handler.py:finish():866] shutting down handler | |
| 2024-04-09 21:55:04,577 INFO WriterThread:160 [datastore.py:close():296] close: /kaggle/working/wandb/run-20240409_214603-e7cki9vp/run-e7cki9vp.wandb | |
| 2024-04-09 21:55:04,648 INFO SenderThread:160 [sender.py:finish():1546] shutting down sender | |
| 2024-04-09 21:55:04,648 INFO SenderThread:160 [file_pusher.py:finish():172] shutting down file pusher | |
| 2024-04-09 21:55:04,648 INFO SenderThread:160 [file_pusher.py:join():178] waiting for file pusher | |