File size: 35,138 Bytes
291544f be9d187 291544f 16d7048 d809bf5 f964374 a8d3c55 50d2747 c743918 50857db edd2417 9208ac0 357d630 70ae419 d71f063 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 |
2024-04-14 08:08:44,754 INFO StreamThr :143 [internal.py:wandb_internal():86] W&B internal server running at pid: 143, started at: 2024-04-14 08:08:44.754030
2024-04-14 08:08:44,756 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status
2024-04-14 08:08:45,288 INFO WriterThread:143 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/run-fw7v7n5v.wandb
2024-04-14 08:08:45,288 DEBUG SenderThread:143 [sender.py:send():379] send: header
2024-04-14 08:08:45,297 DEBUG SenderThread:143 [sender.py:send():379] send: run
2024-04-14 08:08:45,457 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: check_version
2024-04-14 08:08:45,459 INFO SenderThread:143 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files
2024-04-14 08:08:45,459 INFO SenderThread:143 [sender.py:_start_run_threads():1124] run started: fw7v7n5v with start time 1713082124.756758
2024-04-14 08:08:45,459 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: check_version
2024-04-14 08:08:45,554 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: run_start
2024-04-14 08:08:45,566 DEBUG HandlerThread:143 [system_info.py:__init__():26] System info init
2024-04-14 08:08:45,566 DEBUG HandlerThread:143 [system_info.py:__init__():41] System info init done
2024-04-14 08:08:45,566 INFO HandlerThread:143 [system_monitor.py:start():194] Starting system monitor
2024-04-14 08:08:45,566 INFO SystemMonitor:143 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-14 08:08:45,566 INFO HandlerThread:143 [system_monitor.py:probe():214] Collecting system info
2024-04-14 08:08:45,567 INFO SystemMonitor:143 [interfaces.py:start():190] Started cpu monitoring
2024-04-14 08:08:45,568 INFO SystemMonitor:143 [interfaces.py:start():190] Started disk monitoring
2024-04-14 08:08:45,568 INFO SystemMonitor:143 [interfaces.py:start():190] Started gpu monitoring
2024-04-14 08:08:45,569 INFO SystemMonitor:143 [interfaces.py:start():190] Started memory monitoring
2024-04-14 08:08:45,570 INFO SystemMonitor:143 [interfaces.py:start():190] Started network monitoring
2024-04-14 08:08:45,581 DEBUG HandlerThread:143 [system_info.py:probe():150] Probing system
2024-04-14 08:08:45,583 DEBUG HandlerThread:143 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-14 08:08:45,583 DEBUG HandlerThread:143 [system_info.py:probe():198] Probing system done
2024-04-14 08:08:45,583 DEBUG HandlerThread:143 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-14T08:08:45.581597', 'startedAt': '2024-04-14T08:08:44.747647', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '22f82e031f6d', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.188, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5574.93864440918}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-14 08:08:45,583 INFO HandlerThread:143 [system_monitor.py:probe():224] Finished collecting system info
2024-04-14 08:08:45,583 INFO HandlerThread:143 [system_monitor.py:probe():227] Publishing system info
2024-04-14 08:08:45,583 DEBUG HandlerThread:143 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-14 08:08:46,461 INFO Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/conda-environment.yaml
2024-04-14 08:09:00,599 ERROR HandlerThread:143 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
subprocess.call(
File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
return p.wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
return self._wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-14 08:09:00,605 DEBUG HandlerThread:143 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-14 08:09:00,605 INFO HandlerThread:143 [system_monitor.py:probe():229] Finished publishing system info
2024-04-14 08:09:00,616 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:00,616 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 08:09:00,616 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:00,616 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 08:09:00,616 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:00,616 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: keepalive
2024-04-14 08:09:00,617 DEBUG SenderThread:143 [sender.py:send():379] send: files
2024-04-14 08:09:00,617 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-14 08:09:00,816 INFO wandb-upload_0:143 [upload_job.py:push():131] Uploaded file /tmp/tmp6ddeqyu6wandb/52qsjx8f-wandb-metadata.json
2024-04-14 08:09:01,463 INFO Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-metadata.json
2024-04-14 08:09:01,703 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: python_packages
2024-04-14 08:09:01,704 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: python_packages
2024-04-14 08:09:01,708 DEBUG SenderThread:143 [sender.py:send():379] send: telemetry
2024-04-14 08:09:01,709 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:01,709 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:01,719 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:01,804 DEBUG SenderThread:143 [sender.py:send():379] send: config
2024-04-14 08:09:01,807 DEBUG SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:01,807 DEBUG SenderThread:143 [sender.py:send():379] send: telemetry
2024-04-14 08:09:01,807 DEBUG SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:01,807 WARNING SenderThread:143 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-14 08:09:01,807 DEBUG SenderThread:143 [sender.py:send():379] send: telemetry
2024-04-14 08:09:02,463 INFO Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/requirements.txt
2024-04-14 08:09:03,464 INFO Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:09:05,465 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:09:05,923 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:10,924 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:15,929 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:16,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:16,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:16,814 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:17,470 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/config.yaml
2024-04-14 08:09:21,866 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:26,867 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:31,704 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:31,705 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:31,745 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:32,782 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:37,783 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:42,784 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:45,570 DEBUG SystemMonitor:143 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-14 08:09:45,572 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:09:46,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:09:46,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:09:46,747 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:09:48,774 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:52,118 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:09:52,121 DEBUG SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG SenderThread:143 [sender.py:send():379] send: metric
2024-04-14 08:09:52,121 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:09:52,121 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:09:52,122 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:09:52,483 INFO Thread-12 :143 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:09:53,937 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:09:55,484 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:09:58,938 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:01,705 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:01,705 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:01,709 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:04,792 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:09,793 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:14,794 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:15,573 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:10:16,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:16,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:16,748 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:20,779 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:21,494 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/config.yaml
2024-04-14 08:10:25,885 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:30,885 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:31,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:31,708 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:31,748 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:36,803 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:41,803 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:43,062 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:10:43,063 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:10:43,063 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:10:43,065 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:10:43,502 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:10:45,503 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:10:45,574 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:10:46,705 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:10:46,706 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:10:46,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:10:46,840 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:51,840 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:10:56,841 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:01,706 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:01,706 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:01,746 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:02,799 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:03,942 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:11:03,944 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:11:03,944 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:11:03,944 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:11:04,511 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:11:07,512 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:11:08,648 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:13,649 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:15,575 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:11:16,717 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:16,717 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:16,718 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:18,816 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:23,817 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:24,756 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:11:24,757 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:11:24,757 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:11:24,759 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:11:25,520 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:11:27,521 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:11:29,691 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:31,706 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:31,706 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:31,709 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:34,743 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:39,744 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:44,745 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:45,576 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:11:46,706 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:11:46,706 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:11:46,747 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:11:49,817 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:11:50,426 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:11:50,427 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:11:50,427 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:11:50,429 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:11:50,531 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:11:53,533 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:11:55,104 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:00,105 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:01,706 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:01,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:01,709 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:05,770 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:10,772 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:12,237 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:12:12,238 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:12:12,239 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:12:12,239 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:12:12,542 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:12:15,544 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:12:15,576 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:12:16,577 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:16,706 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:16,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:16,709 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:21,837 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:26,838 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:31,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:31,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:31,747 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:32,804 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:37,569 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:12:37,570 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:12:37,570 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:12:37,572 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:12:38,260 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:38,553 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:12:39,554 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:12:43,262 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:45,577 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:12:46,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:12:46,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:12:46,710 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:12:48,814 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:53,814 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:12:58,316 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:12:58,317 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:12:58,318 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:12:58,318 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:12:58,562 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:12:58,997 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:01,564 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:13:01,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:01,707 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:01,710 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:04,731 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:09,731 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:14,733 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:15,578 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:13:16,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:16,708 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:16,748 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:19,828 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:24,829 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:25,132 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:13:25,134 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:13:25,134 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:13:25,136 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:13:25,574 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:13:27,575 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:13:30,810 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:31,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:31,708 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:31,710 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:36,736 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:41,737 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:45,579 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:13:45,910 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:13:45,911 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:13:45,912 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:13:45,912 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:13:46,582 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:13:46,707 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:13:46,708 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:13:46,708 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:13:46,796 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:47,582 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:13:51,797 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:13:56,797 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:01,708 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:01,709 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:01,749 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:02,767 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:06,631 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:14:06,632 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:14:06,633 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:14:06,633 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:14:07,589 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:14:08,286 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:09,590 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:14:13,287 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:15,580 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:14:16,708 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:16,708 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:16,709 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:18,798 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:23,799 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:27,910 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:14:27,913 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:14:27,913 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:14:27,914 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:14:28,597 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
2024-04-14 08:14:29,587 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:29,597 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/output.log
2024-04-14 08:14:31,708 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:31,709 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:31,712 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:34,830 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:39,831 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:44,832 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: status_report
2024-04-14 08:14:45,581 DEBUG SenderThread:143 [sender.py:send():379] send: stats
2024-04-14 08:14:46,708 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: stop_status
2024-04-14 08:14:46,709 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: stop_status
2024-04-14 08:14:46,749 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-14 08:14:48,932 DEBUG HandlerThread:143 [handler.py:handle_request():146] handle_request: partial_history
2024-04-14 08:14:48,933 DEBUG SenderThread:143 [sender.py:send():379] send: history
2024-04-14 08:14:48,934 DEBUG SenderThread:143 [sender.py:send_request():406] send_request: summary_record
2024-04-14 08:14:48,934 INFO SenderThread:143 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-14 08:14:49,605 INFO Thread-12 :143 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240414_080844-fw7v7n5v/files/wandb-summary.json
|