File size: 17,601 Bytes
60aab60 13742f4 95270cd f21684b 04de011 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 |
2024-04-14 07:31:17,436 INFO StreamThr :183 [internal.py:wandb_internal():86] W&B internal server running at pid: 183, started at: 2024-04-14 07:31:17.436199
2024-04-14 07:31:17,438 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status
2024-04-14 07:31:17,994 INFO WriterThread:183 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/run-xuqlocdo.wandb
2024-04-14 07:31:17,994 DEBUG SenderThread:183 [sender.py:send():379] send: header
2024-04-14 07:31:17,998 DEBUG SenderThread:183 [sender.py:send():379] send: run
2024-04-14 07:31:18,114 INFO SenderThread:183 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files
2024-04-14 07:31:18,115 INFO SenderThread:183 [sender.py:_start_run_threads():1124] run started: xuqlocdo with start time 1713079877.439223
2024-04-14 07:31:18,123 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: check_version
2024-04-14 07:31:18,123 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: check_version
2024-04-14 07:31:18,219 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: run_start
2024-04-14 07:31:18,230 DEBUG HandlerThread:183 [system_info.py:__init__():26] System info init
2024-04-14 07:31:18,230 DEBUG HandlerThread:183 [system_info.py:__init__():41] System info init done
2024-04-14 07:31:18,230 INFO HandlerThread:183 [system_monitor.py:start():194] Starting system monitor
2024-04-14 07:31:18,230 INFO SystemMonitor:183 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-14 07:31:18,231 INFO HandlerThread:183 [system_monitor.py:probe():214] Collecting system info
2024-04-14 07:31:18,231 INFO SystemMonitor:183 [interfaces.py:start():190] Started cpu monitoring
2024-04-14 07:31:18,232 INFO SystemMonitor:183 [interfaces.py:start():190] Started disk monitoring
2024-04-14 07:31:18,233 INFO SystemMonitor:183 [interfaces.py:start():190] Started gpu monitoring
2024-04-14 07:31:18,234 INFO SystemMonitor:183 [interfaces.py:start():190] Started memory monitoring
2024-04-14 07:31:18,235 INFO SystemMonitor:183 [interfaces.py:start():190] Started network monitoring
2024-04-14 07:31:18,245 DEBUG HandlerThread:183 [system_info.py:probe():150] Probing system
2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [system_info.py:probe():198] Probing system done
2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-14T07:31:18.245345', 'startedAt': '2024-04-14T07:31:17.430147', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'f694866fb244', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.172, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.172, 'min': 0.0, 'max': 0.0}, {'current': 2000.172, 'min': 0.0, 'max': 0.0}, {'current': 2000.172, 'min': 0.0, 'max': 0.0}, {'current': 2000.172, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5574.948589324951}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-14 07:31:18,247 INFO HandlerThread:183 [system_monitor.py:probe():224] Finished collecting system info
2024-04-14 07:31:18,247 INFO HandlerThread:183 [system_monitor.py:probe():227] Publishing system info
2024-04-14 07:31:18,247 DEBUG HandlerThread:183 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-14 07:31:19,116 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/conda-environment.yaml
2024-04-14 07:31:33,262 ERROR HandlerThread:183 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
subprocess.call(
File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
return p.wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
return self._wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-14 07:31:33,268 DEBUG HandlerThread:183 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-14 07:31:33,268 INFO HandlerThread:183 [system_monitor.py:probe():229] Finished publishing system info
2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 07:31:33,278 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:33,279 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 07:31:33,279 DEBUG SenderThread:183 [sender.py:send():379] send: files
2024-04-14 07:31:33,279 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-14 07:31:33,552 INFO wandb-upload_0:183 [upload_job.py:push():131] Uploaded file /tmp/tmpbxtmxuylwandb/9u0zqbnq-wandb-metadata.json
2024-04-14 07:31:34,120 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-metadata.json
2024-04-14 07:31:34,297 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: python_packages
2024-04-14 07:31:34,297 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: python_packages
2024-04-14 07:31:34,301 DEBUG SenderThread:183 [sender.py:send():379] send: telemetry
2024-04-14 07:31:34,311 DEBUG SenderThread:183 [sender.py:send():379] send: config
2024-04-14 07:31:34,312 DEBUG SenderThread:183 [sender.py:send():379] send: metric
2024-04-14 07:31:34,313 DEBUG SenderThread:183 [sender.py:send():379] send: telemetry
2024-04-14 07:31:34,313 DEBUG SenderThread:183 [sender.py:send():379] send: metric
2024-04-14 07:31:34,313 WARNING SenderThread:183 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-14 07:31:34,313 DEBUG SenderThread:183 [sender.py:send():379] send: telemetry
2024-04-14 07:31:34,314 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:31:34,314 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:31:34,317 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:31:35,120 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/requirements.txt
2024-04-14 07:31:36,120 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log
2024-04-14 07:31:38,121 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log
2024-04-14 07:31:38,719 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:43,720 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:48,726 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:49,125 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/config.yaml
2024-04-14 07:31:49,299 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:31:49,300 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:31:49,300 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:31:54,400 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:31:56,464 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 07:31:56,465 DEBUG SenderThread:183 [sender.py:send():379] send: metric
2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: metric
2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: metric
2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: metric
2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send():379] send: history
2024-04-14 07:31:56,466 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record
2024-04-14 07:31:56,468 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 07:31:57,129 INFO Thread-12 :183 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json
2024-04-14 07:32:00,130 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log
2024-04-14 07:32:00,285 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:04,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:32:04,298 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:32:04,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:32:05,328 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:10,328 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:15,329 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:17,474 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 07:32:17,475 DEBUG SenderThread:183 [sender.py:send():379] send: history
2024-04-14 07:32:17,475 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record
2024-04-14 07:32:17,477 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 07:32:18,137 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json
2024-04-14 07:32:18,235 DEBUG SystemMonitor:183 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-14 07:32:18,237 DEBUG SenderThread:183 [sender.py:send():379] send: stats
2024-04-14 07:32:19,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:32:19,298 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:32:19,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:32:20,138 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log
2024-04-14 07:32:20,418 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:21,138 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/config.yaml
2024-04-14 07:32:25,504 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:30,505 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:34,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:32:34,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:32:34,338 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:32:36,378 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:38,751 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 07:32:38,753 DEBUG SenderThread:183 [sender.py:send():379] send: history
2024-04-14 07:32:38,753 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record
2024-04-14 07:32:38,753 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 07:32:39,145 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json
2024-04-14 07:32:41,573 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:42,147 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log
2024-04-14 07:32:46,574 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:48,238 DEBUG SenderThread:183 [sender.py:send():379] send: stats
2024-04-14 07:32:49,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:32:49,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:32:49,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:32:52,417 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:57,418 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:32:59,581 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 07:32:59,582 DEBUG SenderThread:183 [sender.py:send():379] send: history
2024-04-14 07:32:59,582 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record
2024-04-14 07:32:59,582 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 07:33:00,154 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json
2024-04-14 07:33:02,155 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/output.log
2024-04-14 07:33:03,319 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:33:04,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:33:04,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:33:04,301 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:33:08,369 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:33:13,370 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:33:18,240 DEBUG SenderThread:183 [sender.py:send():379] send: stats
2024-04-14 07:33:19,241 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 07:33:19,298 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 07:33:19,299 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: stop_status
2024-04-14 07:33:19,339 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 07:33:22,849 DEBUG HandlerThread:183 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 07:33:22,850 DEBUG SenderThread:183 [sender.py:send():379] send: history
2024-04-14 07:33:22,851 DEBUG SenderThread:183 [sender.py:send_request():406] send_request: summary_record
2024-04-14 07:33:22,852 INFO SenderThread:183 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 07:33:23,163 INFO Thread-12 :183 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_073117-xuqlocdo/files/wandb-summary.json
|