File size: 25,929 Bytes
c59484b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 |
2024-04-10 11:42:43,188 INFO StreamThr :144 [internal.py:wandb_internal():86] W&B internal server running at pid: 144, started at: 2024-04-10 11:42:43.187267
2024-04-10 11:42:43,189 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status
2024-04-10 11:42:43,491 INFO WriterThread:144 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/run-a6hgqntd.wandb
2024-04-10 11:42:43,492 DEBUG SenderThread:144 [sender.py:send():379] send: header
2024-04-10 11:42:43,495 DEBUG SenderThread:144 [sender.py:send():379] send: run
2024-04-10 11:42:43,678 INFO SenderThread:144 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files
2024-04-10 11:42:43,678 INFO SenderThread:144 [sender.py:_start_run_threads():1124] run started: a6hgqntd with start time 1712749363.188282
2024-04-10 11:42:43,686 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: check_version
2024-04-10 11:42:43,686 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: check_version
2024-04-10 11:42:43,780 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: run_start
2024-04-10 11:42:43,792 DEBUG HandlerThread:144 [system_info.py:__init__():26] System info init
2024-04-10 11:42:43,792 DEBUG HandlerThread:144 [system_info.py:__init__():41] System info init done
2024-04-10 11:42:43,792 INFO HandlerThread:144 [system_monitor.py:start():194] Starting system monitor
2024-04-10 11:42:43,792 INFO SystemMonitor:144 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-10 11:42:43,792 INFO HandlerThread:144 [system_monitor.py:probe():214] Collecting system info
2024-04-10 11:42:43,793 INFO SystemMonitor:144 [interfaces.py:start():190] Started cpu monitoring
2024-04-10 11:42:43,793 INFO SystemMonitor:144 [interfaces.py:start():190] Started disk monitoring
2024-04-10 11:42:43,794 INFO SystemMonitor:144 [interfaces.py:start():190] Started gpu monitoring
2024-04-10 11:42:43,795 INFO SystemMonitor:144 [interfaces.py:start():190] Started memory monitoring
2024-04-10 11:42:43,796 INFO SystemMonitor:144 [interfaces.py:start():190] Started network monitoring
2024-04-10 11:42:43,806 DEBUG HandlerThread:144 [system_info.py:probe():150] Probing system
2024-04-10 11:42:43,808 DEBUG HandlerThread:144 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-10 11:42:43,808 DEBUG HandlerThread:144 [system_info.py:probe():198] Probing system done
2024-04-10 11:42:43,808 DEBUG HandlerThread:144 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-10T11:42:43.806558', 'startedAt': '2024-04-10T11:42:43.180890', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '57bca537cf5f', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5568.826057434082}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-10 11:42:43,809 INFO HandlerThread:144 [system_monitor.py:probe():224] Finished collecting system info
2024-04-10 11:42:43,809 INFO HandlerThread:144 [system_monitor.py:probe():227] Publishing system info
2024-04-10 11:42:43,809 DEBUG HandlerThread:144 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-10 11:42:44,682 INFO Thread-12 :144 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/conda-environment.yaml
2024-04-10 11:42:58,823 ERROR HandlerThread:144 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
subprocess.call(
File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
return p.wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
return self._wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-10 11:42:58,825 DEBUG HandlerThread:144 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-10 11:42:58,825 INFO HandlerThread:144 [system_monitor.py:probe():229] Finished publishing system info
2024-04-10 11:42:58,832 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:42:58,833 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: keepalive
2024-04-10 11:42:58,833 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:42:58,833 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: keepalive
2024-04-10 11:42:58,833 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:42:58,833 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: keepalive
2024-04-10 11:42:58,834 DEBUG SenderThread:144 [sender.py:send():379] send: files
2024-04-10 11:42:58,834 INFO SenderThread:144 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-10 11:42:59,058 INFO wandb-upload_0:144 [upload_job.py:push():131] Uploaded file /tmp/tmp3lpmf8m8wandb/q0pbe4nu-wandb-metadata.json
2024-04-10 11:42:59,684 INFO Thread-12 :144 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/wandb-metadata.json
2024-04-10 11:42:59,776 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: python_packages
2024-04-10 11:42:59,776 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: python_packages
2024-04-10 11:42:59,780 DEBUG SenderThread:144 [sender.py:send():379] send: telemetry
2024-04-10 11:42:59,786 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 11:42:59,786 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: stop_status
2024-04-10 11:42:59,790 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 11:42:59,894 DEBUG SenderThread:144 [sender.py:send():379] send: config
2024-04-10 11:42:59,896 DEBUG SenderThread:144 [sender.py:send():379] send: metric
2024-04-10 11:42:59,896 DEBUG SenderThread:144 [sender.py:send():379] send: telemetry
2024-04-10 11:42:59,897 DEBUG SenderThread:144 [sender.py:send():379] send: metric
2024-04-10 11:42:59,897 WARNING SenderThread:144 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-10 11:42:59,897 DEBUG SenderThread:144 [sender.py:send():379] send: telemetry
2024-04-10 11:43:00,684 INFO Thread-12 :144 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/output.log
2024-04-10 11:43:00,685 INFO Thread-12 :144 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/requirements.txt
2024-04-10 11:43:02,685 INFO Thread-12 :144 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/output.log
2024-04-10 11:43:03,929 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:08,930 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:13,936 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:14,690 INFO Thread-12 :144 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/config.yaml
2024-04-10 11:43:14,779 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 11:43:14,779 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 11:43:14,780 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: stop_status
2024-04-10 11:43:19,838 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:24,839 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:29,777 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 11:43:29,778 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: stop_status
2024-04-10 11:43:29,817 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 11:43:30,827 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:35,828 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:40,829 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:43,736 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: pause
2024-04-10 11:43:43,736 INFO HandlerThread:144 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-04-10 11:43:43,736 INFO HandlerThread:144 [system_monitor.py:finish():203] Stopping system monitor
2024-04-10 11:43:43,736 DEBUG SystemMonitor:144 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-10 11:43:43,737 DEBUG SystemMonitor:144 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-04-10 11:43:43,737 DEBUG SystemMonitor:144 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-04-10 11:43:43,738 INFO HandlerThread:144 [interfaces.py:finish():202] Joined cpu monitor
2024-04-10 11:43:43,738 INFO HandlerThread:144 [interfaces.py:finish():202] Joined disk monitor
2024-04-10 11:43:43,747 INFO HandlerThread:144 [interfaces.py:finish():202] Joined gpu monitor
2024-04-10 11:43:43,747 INFO HandlerThread:144 [interfaces.py:finish():202] Joined memory monitor
2024-04-10 11:43:43,748 INFO HandlerThread:144 [interfaces.py:finish():202] Joined network monitor
2024-04-10 11:43:43,748 DEBUG SenderThread:144 [sender.py:send():379] send: stats
2024-04-10 11:43:44,777 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 11:43:44,778 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: stop_status
2024-04-10 11:43:44,778 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 11:43:46,829 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:51,830 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:56,832 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:57,053 DEBUG SenderThread:144 [sender.py:send():379] send: exit
2024-04-10 11:43:57,053 INFO SenderThread:144 [sender.py:send_exit():586] handling exit code: 0
2024-04-10 11:43:57,053 INFO SenderThread:144 [sender.py:send_exit():588] handling runtime: 59
2024-04-10 11:43:57,055 INFO SenderThread:144 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 11:43:57,055 INFO SenderThread:144 [sender.py:send_exit():594] send defer
2024-04-10 11:43:57,055 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,055 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 0
2024-04-10 11:43:57,056 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,056 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 0
2024-04-10 11:43:57,056 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 1
2024-04-10 11:43:57,056 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,056 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 1
2024-04-10 11:43:57,056 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,056 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 1
2024-04-10 11:43:57,056 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 2
2024-04-10 11:43:57,056 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,056 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 2
2024-04-10 11:43:57,057 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,057 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 2
2024-04-10 11:43:57,057 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 3
2024-04-10 11:43:57,057 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,057 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 3
2024-04-10 11:43:57,057 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,057 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 3
2024-04-10 11:43:57,057 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 4
2024-04-10 11:43:57,057 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,057 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 4
2024-04-10 11:43:57,058 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,058 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 4
2024-04-10 11:43:57,058 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 5
2024-04-10 11:43:57,058 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,058 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 5
2024-04-10 11:43:57,058 DEBUG SenderThread:144 [sender.py:send():379] send: summary
2024-04-10 11:43:57,058 INFO SenderThread:144 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 11:43:57,059 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,059 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 5
2024-04-10 11:43:57,059 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 6
2024-04-10 11:43:57,059 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,059 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 6
2024-04-10 11:43:57,059 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,059 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 6
2024-04-10 11:43:57,059 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 7
2024-04-10 11:43:57,059 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 11:43:57,059 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,059 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 7
2024-04-10 11:43:57,060 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,060 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 7
2024-04-10 11:43:57,707 INFO Thread-12 :144 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/wandb-summary.json
2024-04-10 11:43:57,970 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 8
2024-04-10 11:43:57,971 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,971 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 8
2024-04-10 11:43:57,971 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,971 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 8
2024-04-10 11:43:57,971 INFO SenderThread:144 [job_builder.py:build():318] Attempting to build job artifact
2024-04-10 11:43:57,973 INFO SenderThread:144 [job_builder.py:_get_source_type():466] no source found
2024-04-10 11:43:57,973 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 9
2024-04-10 11:43:57,973 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:57,973 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 9
2024-04-10 11:43:57,974 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:57,974 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 9
2024-04-10 11:43:57,974 INFO SenderThread:144 [dir_watcher.py:finish():358] shutting down directory watcher
2024-04-10 11:43:58,053 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: poll_exit
2024-04-10 11:43:58,707 INFO Thread-12 :144 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/output.log
2024-04-10 11:43:58,708 INFO SenderThread:144 [dir_watcher.py:finish():388] scan: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files
2024-04-10 11:43:58,708 INFO SenderThread:144 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/config.yaml config.yaml
2024-04-10 11:43:58,708 INFO SenderThread:144 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/conda-environment.yaml conda-environment.yaml
2024-04-10 11:43:58,711 INFO SenderThread:144 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/output.log output.log
2024-04-10 11:43:58,712 INFO SenderThread:144 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/wandb-metadata.json wandb-metadata.json
2024-04-10 11:43:58,712 INFO SenderThread:144 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/requirements.txt requirements.txt
2024-04-10 11:43:58,712 INFO SenderThread:144 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/wandb-summary.json wandb-summary.json
2024-04-10 11:43:58,715 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 10
2024-04-10 11:43:58,716 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: poll_exit
2024-04-10 11:43:58,718 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:58,719 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 10
2024-04-10 11:43:58,722 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:58,722 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 10
2024-04-10 11:43:58,722 INFO SenderThread:144 [file_pusher.py:finish():172] shutting down file pusher
2024-04-10 11:43:58,867 INFO wandb-upload_0:144 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/config.yaml
2024-04-10 11:43:58,941 INFO wandb-upload_1:144 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/output.log
2024-04-10 11:43:58,948 INFO wandb-upload_3:144 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/wandb-summary.json
2024-04-10 11:43:58,986 INFO wandb-upload_2:144 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240410_114243-a6hgqntd/files/requirements.txt
2024-04-10 11:43:59,055 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: poll_exit
2024-04-10 11:43:59,055 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: poll_exit
2024-04-10 11:43:59,186 INFO Thread-11 (_thread_body):144 [sender.py:transition_state():614] send defer: 11
2024-04-10 11:43:59,187 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:59,187 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 11
2024-04-10 11:43:59,187 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:59,187 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 11
2024-04-10 11:43:59,187 INFO SenderThread:144 [file_pusher.py:join():178] waiting for file pusher
2024-04-10 11:43:59,188 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 12
2024-04-10 11:43:59,188 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:59,188 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 12
2024-04-10 11:43:59,188 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:59,189 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 12
2024-04-10 11:43:59,189 INFO SenderThread:144 [file_stream.py:finish():614] file stream finish called
2024-04-10 11:43:59,266 INFO SenderThread:144 [file_stream.py:finish():618] file stream finish is done
2024-04-10 11:43:59,266 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 13
2024-04-10 11:43:59,266 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:59,266 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 13
2024-04-10 11:43:59,266 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:59,266 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 13
2024-04-10 11:43:59,266 INFO SenderThread:144 [sender.py:transition_state():614] send defer: 14
2024-04-10 11:43:59,267 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: defer
2024-04-10 11:43:59,267 INFO HandlerThread:144 [handler.py:handle_request_defer():172] handle defer: 14
2024-04-10 11:43:59,267 DEBUG SenderThread:144 [sender.py:send():379] send: final
2024-04-10 11:43:59,267 DEBUG SenderThread:144 [sender.py:send():379] send: footer
2024-04-10 11:43:59,267 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: defer
2024-04-10 11:43:59,268 INFO SenderThread:144 [sender.py:send_request_defer():610] handle sender defer: 14
2024-04-10 11:43:59,268 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: poll_exit
2024-04-10 11:43:59,269 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: poll_exit
2024-04-10 11:43:59,269 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: poll_exit
2024-04-10 11:43:59,270 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: poll_exit
2024-04-10 11:43:59,270 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: server_info
2024-04-10 11:43:59,271 DEBUG SenderThread:144 [sender.py:send_request():406] send_request: server_info
2024-04-10 11:43:59,275 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: get_summary
2024-04-10 11:43:59,275 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: sampled_history
2024-04-10 11:43:59,276 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 11:43:59,289 INFO MainThread:144 [wandb_run.py:_footer_history_summary_info():3920] rendering history
2024-04-10 11:43:59,289 INFO MainThread:144 [wandb_run.py:_footer_history_summary_info():3952] rendering summary
2024-04-10 11:43:59,289 INFO MainThread:144 [wandb_run.py:_footer_sync_info():3879] logging synced files
2024-04-10 11:43:59,289 DEBUG HandlerThread:144 [handler.py:handle_request():146] handle_request: shutdown
2024-04-10 11:43:59,290 INFO HandlerThread:144 [handler.py:finish():866] shutting down handler
2024-04-10 11:44:00,271 INFO WriterThread:144 [datastore.py:close():296] close: /kaggle/working/wandb/run-20240410_114243-a6hgqntd/run-a6hgqntd.wandb
2024-04-10 11:44:00,289 INFO SenderThread:144 [sender.py:finish():1546] shutting down sender
2024-04-10 11:44:00,289 INFO SenderThread:144 [file_pusher.py:finish():172] shutting down file pusher
2024-04-10 11:44:00,289 INFO SenderThread:144 [file_pusher.py:join():178] waiting for file pusher
|