File size: 17,158 Bytes
0f88c91 be9d187 0f88c91 7e674a5 720031a 333d6bf 1fcd4be 385a0c1 683f65c 0cc7940 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
2024-04-12 07:35:55,749 INFO StreamThr :334 [internal.py:wandb_internal():86] W&B internal server running at pid: 334, started at: 2024-04-12 07:35:55.748821
2024-04-12 07:35:55,751 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status
2024-04-12 07:35:56,159 INFO WriterThread:334 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/run-bw7oy9ix.wandb
2024-04-12 07:35:56,160 DEBUG SenderThread:334 [sender.py:send():379] send: header
2024-04-12 07:35:56,163 DEBUG SenderThread:334 [sender.py:send():379] send: run
2024-04-12 07:35:56,307 INFO SenderThread:334 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files
2024-04-12 07:35:56,307 INFO SenderThread:334 [sender.py:_start_run_threads():1124] run started: bw7oy9ix with start time 1712907355.748886
2024-04-12 07:35:56,317 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: check_version
2024-04-12 07:35:56,317 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: check_version
2024-04-12 07:35:56,409 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: run_start
2024-04-12 07:35:56,421 DEBUG HandlerThread:334 [system_info.py:__init__():26] System info init
2024-04-12 07:35:56,421 DEBUG HandlerThread:334 [system_info.py:__init__():41] System info init done
2024-04-12 07:35:56,422 INFO HandlerThread:334 [system_monitor.py:start():194] Starting system monitor
2024-04-12 07:35:56,422 INFO SystemMonitor:334 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-12 07:35:56,422 INFO HandlerThread:334 [system_monitor.py:probe():214] Collecting system info
2024-04-12 07:35:56,423 INFO SystemMonitor:334 [interfaces.py:start():190] Started cpu monitoring
2024-04-12 07:35:56,423 INFO SystemMonitor:334 [interfaces.py:start():190] Started disk monitoring
2024-04-12 07:35:56,424 INFO SystemMonitor:334 [interfaces.py:start():190] Started gpu monitoring
2024-04-12 07:35:56,425 INFO SystemMonitor:334 [interfaces.py:start():190] Started memory monitoring
2024-04-12 07:35:56,426 INFO SystemMonitor:334 [interfaces.py:start():190] Started network monitoring
2024-04-12 07:35:56,437 DEBUG HandlerThread:334 [system_info.py:probe():150] Probing system
2024-04-12 07:35:56,439 DEBUG HandlerThread:334 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-12 07:35:56,439 DEBUG HandlerThread:334 [system_info.py:probe():198] Probing system done
2024-04-12 07:35:56,439 DEBUG HandlerThread:334 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-12T07:35:56.437403', 'startedAt': '2024-04-12T07:35:55.741618', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'e5a48bec8248', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.138, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5565.782459259033}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-12 07:35:56,440 INFO HandlerThread:334 [system_monitor.py:probe():224] Finished collecting system info
2024-04-12 07:35:56,440 INFO HandlerThread:334 [system_monitor.py:probe():227] Publishing system info
2024-04-12 07:35:56,440 DEBUG HandlerThread:334 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-12 07:35:57,309 INFO Thread-12 :334 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/conda-environment.yaml
2024-04-12 07:36:11,455 ERROR HandlerThread:334 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
subprocess.call(
File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
return p.wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
return self._wait(timeout=timeout)
File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-12 07:36:11,456 DEBUG HandlerThread:334 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-12 07:36:11,456 INFO HandlerThread:334 [system_monitor.py:probe():229] Finished publishing system info
2024-04-12 07:36:11,462 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:11,462 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:36:11,462 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:11,462 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:36:11,462 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:11,462 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:36:11,463 DEBUG SenderThread:334 [sender.py:send():379] send: files
2024-04-12 07:36:11,463 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-12 07:36:11,664 INFO wandb-upload_0:334 [upload_job.py:push():131] Uploaded file /tmp/tmpjph8qv3dwandb/f1up76ir-wandb-metadata.json
2024-04-12 07:36:12,312 INFO Thread-12 :334 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-metadata.json
2024-04-12 07:36:12,509 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: python_packages
2024-04-12 07:36:12,509 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: python_packages
2024-04-12 07:36:12,512 DEBUG SenderThread:334 [sender.py:send():379] send: telemetry
2024-04-12 07:36:12,523 DEBUG SenderThread:334 [sender.py:send():379] send: config
2024-04-12 07:36:12,525 DEBUG SenderThread:334 [sender.py:send():379] send: metric
2024-04-12 07:36:12,534 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:36:12,535 DEBUG SenderThread:334 [sender.py:send():379] send: telemetry
2024-04-12 07:36:12,535 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:36:12,535 DEBUG SenderThread:334 [sender.py:send():379] send: metric
2024-04-12 07:36:12,536 WARNING SenderThread:334 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-12 07:36:12,536 DEBUG SenderThread:334 [sender.py:send():379] send: telemetry
2024-04-12 07:36:12,536 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:36:13,313 INFO Thread-12 :334 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/requirements.txt
2024-04-12 07:36:13,313 INFO Thread-12 :334 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:15,314 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:17,014 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:19,589 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:19,592 DEBUG SenderThread:334 [sender.py:send():379] send: metric
2024-04-12 07:36:19,593 DEBUG SenderThread:334 [sender.py:send():379] send: metric
2024-04-12 07:36:19,593 DEBUG SenderThread:334 [sender.py:send():379] send: metric
2024-04-12 07:36:19,593 DEBUG SenderThread:334 [sender.py:send():379] send: metric
2024-04-12 07:36:19,593 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:19,593 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:19,595 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:36:20,316 INFO Thread-12 :334 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:36:21,316 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:22,300 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:26,097 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:26,098 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:26,098 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:26,100 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:36:26,318 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:36:27,510 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:36:27,511 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:36:27,511 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:36:27,582 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:28,319 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/config.yaml
2024-04-12 07:36:29,319 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:32,526 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:32,527 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:32,527 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:32,530 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:36:33,242 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:33,320 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:36:35,321 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:38,243 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:39,263 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:39,264 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:39,265 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:39,267 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:36:39,323 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:36:41,324 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:42,512 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:36:42,512 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:36:42,513 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:36:43,604 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:46,420 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:46,421 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:46,422 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:46,424 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:36:47,326 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:36:49,121 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:49,327 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:53,592 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:53,593 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:53,593 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:53,595 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:36:54,299 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:54,329 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:36:55,330 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:36:56,426 DEBUG SystemMonitor:334 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-12 07:36:56,427 DEBUG SenderThread:334 [sender.py:send():379] send: stats
2024-04-12 07:36:57,510 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:36:57,511 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:36:57,514 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:36:59,556 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:36:59,727 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:36:59,728 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:36:59,729 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:36:59,729 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:37:00,332 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
2024-04-12 07:37:03,333 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/output.log
2024-04-12 07:37:04,722 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:37:06,767 DEBUG HandlerThread:334 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:37:06,769 DEBUG SenderThread:334 [sender.py:send():379] send: history
2024-04-12 07:37:06,769 DEBUG SenderThread:334 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:37:06,772 INFO SenderThread:334 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:37:07,335 INFO Thread-12 :334 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_073555-bw7oy9ix/files/wandb-summary.json
|