| 2024-04-14 07:31:17,436 INFO StreamThr :183 [internal.py:wandb_internal():86] W&B internal server running at pid: 183, started at: 2024-04-14 07:31:17.436199 | |
| 2024-04-14 07:31:17,438 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status | |
| 2024-04-14 07:31:17,994 INFO WriterThread:183 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/run-xuqlocdo.wandb | |
| 2024-04-14 07:31:17,994 DEBUG SenderThread:183 [sender.py:send():379] send: header | |
| 2024-04-14 07:31:17,998 DEBUG SenderThread:183 [sender.py:send():379] send: run | |
| 2024-04-14 07:31:18,114 INFO SenderThread:183 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files | |
| 2024-04-14 07:31:18,115 INFO SenderThread:183 [sender.py:_start_run_threads():1124] run started: xuqlocdo with start time 1713079877.439223 | |
| 2024-04-14 07:31:18,123 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-04-14 07:31:18,123 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: check_version | |
| 2024-04-14 07:31:18,219 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-04-14 07:31:18,230 DEBUG HandlerThread:183 [system_info.py:__init__():26] System info init | |
| 2024-04-14 07:31:18,230 DEBUG HandlerThread:183 [system_info.py:__init__():41] System info init done | |
| 2024-04-14 07:31:18,230 INFO HandlerThread:183 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-14 07:31:18,230 INFO SystemMonitor:183 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-14 07:31:18,231 INFO HandlerThread:183 [system_monitor.py:probe():214] Collecting system info | |
| 2024-04-14 07:31:18,231 INFO SystemMonitor:183 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-14 07:31:18,232 INFO SystemMonitor:183 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-14 07:31:18,233 INFO SystemMonitor:183 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-14 07:31:18,234 INFO SystemMonitor:183 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-14 07:31:18,235 INFO SystemMonitor:183 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-14 07:31:18,245 DEBUG HandlerThread:183 [system_info.py:probe():150] Probing system | |
| 2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [system_info.py:probe():198] Probing system done | |
| 2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-14T07:31:18.245345', 'startedAt': '2024-04-14T07:31:17.430147', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'f694866fb244', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.172, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.172, 'min': 0.0, 'max': 0.0}, {'current': 2000.172, 'min': 0.0, 'max': 0.0}, {'current': 2000.172, 'min': 0.0, 'max': 0.0}, {'current': 2000.172, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5574.948589324951}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-04-14 07:31:18,247 INFO HandlerThread:183 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-04-14 07:31:18,247 INFO HandlerThread:183 [system_monitor.py:probe():227] Publishing system info | |
| 2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-04-14 07:31:19,116 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/conda-environment.yaml | |
| 2024-04-14 07:31:33,262 ERROR HandlerThread:183 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds | |
| Traceback (most recent call last): | |
| File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda | |
| subprocess.call( | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call | |
| return p.wait(timeout=timeout) | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait | |
| return self._wait(timeout=timeout) | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait | |
| raise TimeoutExpired(self.args, timeout) | |
| subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds | |
| 2024-04-14 07:31:33,268 DEBUG HandlerThread:183 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-04-14 07:31:33,268 INFO HandlerThread:183 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:33,279 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-14 07:31:33,279 DEBUG SenderThread:183 [sender.py:send():379] send: files | |
| 2024-04-14 07:31:33,279 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now | |
| 2024-04-14 07:31:33,552 INFO wandb-upload_0:183 [upload_job.py:push():131] Uploaded file /tmp/tmpbxtmxuylwandb/9u0zqbnq-wandb-metadata.json | |
| 2024-04-14 07:31:34,120 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-metadata.json | |
| 2024-04-14 07:31:34,297 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-04-14 07:31:34,297 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: python_packages | |
| 2024-04-14 07:31:34,301 DEBUG SenderThread:183 [sender.py:send():379] send: telemetry | |
| 2024-04-14 07:31:34,311 DEBUG SenderThread:183 [sender.py:send():379] send: config | |
| 2024-04-14 07:31:34,312 DEBUG SenderThread:183 [sender.py:send():379] send: metric | |
| 2024-04-14 07:31:34,313 DEBUG SenderThread:183 [sender.py:send():379] send: telemetry | |
| 2024-04-14 07:31:34,313 DEBUG SenderThread:183 [sender.py:send():379] send: metric | |
| 2024-04-14 07:31:34,313 WARNING SenderThread:183 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) | |
| 2024-04-14 07:31:34,313 DEBUG SenderThread:183 [sender.py:send():379] send: telemetry | |
| 2024-04-14 07:31:34,314 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:31:34,314 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:31:34,317 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:31:35,120 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/requirements.txt | |
| 2024-04-14 07:31:36,120 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log | |
| 2024-04-14 07:31:38,121 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log | |
| 2024-04-14 07:31:38,719 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:43,720 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:48,726 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:49,125 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/config.yaml | |
| 2024-04-14 07:31:49,299 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:31:49,300 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:31:49,300 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:31:54,400 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:31:56,464 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-14 07:31:56,465 DEBUG SenderThread:183 [sender.py:send():379] send: metric | |
| 2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: metric | |
| 2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: metric | |
| 2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: metric | |
| 2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: history | |
| 2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-14 07:31:56,468 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-14 07:31:57,129 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json | |
| 2024-04-14 07:32:00,130 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log | |
| 2024-04-14 07:32:00,285 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:04,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:32:04,298 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:32:04,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:32:05,328 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:10,328 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:15,329 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:17,474 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-14 07:32:17,475 DEBUG SenderThread:183 [sender.py:send():379] send: history | |
| 2024-04-14 07:32:17,475 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-14 07:32:17,477 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-14 07:32:18,137 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json | |
| 2024-04-14 07:32:18,235 DEBUG SystemMonitor:183 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-14 07:32:18,237 DEBUG SenderThread:183 [sender.py:send():379] send: stats | |
| 2024-04-14 07:32:19,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:32:19,298 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:32:19,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:32:20,138 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log | |
| 2024-04-14 07:32:20,418 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:21,138 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/config.yaml | |
| 2024-04-14 07:32:25,504 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:30,505 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:34,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:32:34,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:32:34,338 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:32:36,378 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:38,751 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-14 07:32:38,753 DEBUG SenderThread:183 [sender.py:send():379] send: history | |
| 2024-04-14 07:32:38,753 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-14 07:32:38,753 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-14 07:32:39,145 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json | |
| 2024-04-14 07:32:41,573 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:42,147 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log | |
| 2024-04-14 07:32:46,574 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:48,238 DEBUG SenderThread:183 [sender.py:send():379] send: stats | |
| 2024-04-14 07:32:49,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:32:49,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:32:49,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:32:52,417 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:57,418 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:32:59,581 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-14 07:32:59,582 DEBUG SenderThread:183 [sender.py:send():379] send: history | |
| 2024-04-14 07:32:59,582 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-14 07:32:59,582 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-14 07:33:00,154 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json | |
| 2024-04-14 07:33:02,155 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log | |
| 2024-04-14 07:33:03,319 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:33:04,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:33:04,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:33:04,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:33:08,369 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:33:13,370 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:33:18,240 DEBUG SenderThread:183 [sender.py:send():379] send: stats | |
| 2024-04-14 07:33:19,241 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-14 07:33:19,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-14 07:33:19,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-14 07:33:19,339 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-14 07:33:22,849 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-14 07:33:22,850 DEBUG SenderThread:183 [sender.py:send():379] send: history | |
| 2024-04-14 07:33:22,851 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-14 07:33:22,852 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-14 07:33:23,163 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json | |