File size: 19,749 Bytes
8dcd442 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 |
2024-04-08 19:45:11,071 INFO StreamThr :124 [internal.py:wandb_internal():86] W&B internal server running at pid: 124, started at: 2024-04-08 19:45:11.070526
2024-04-08 19:45:11,072 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status
2024-04-08 19:45:11,499 INFO WriterThread:124 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240408_194511-wp046ftq/run-wp046ftq.wandb
2024-04-08 19:45:11,499 DEBUG SenderThread:124 [sender.py:send():378] send: header
2024-04-08 19:45:11,502 DEBUG SenderThread:124 [sender.py:send():378] send: run
2024-04-08 19:45:11,608 INFO SenderThread:124 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files
2024-04-08 19:45:11,608 INFO SenderThread:124 [sender.py:_start_run_threads():1099] run started: wp046ftq with start time 1712605511.071807
2024-04-08 19:45:11,616 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: check_version
2024-04-08 19:45:11,616 DEBUG SenderThread:124 [sender.py:send_request():405] send_request: check_version
2024-04-08 19:45:11,702 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: run_start
2024-04-08 19:45:11,713 DEBUG HandlerThread:124 [system_info.py:__init__():26] System info init
2024-04-08 19:45:11,713 DEBUG HandlerThread:124 [system_info.py:__init__():41] System info init done
2024-04-08 19:45:11,714 INFO HandlerThread:124 [system_monitor.py:start():194] Starting system monitor
2024-04-08 19:45:11,714 INFO SystemMonitor:124 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-08 19:45:11,714 INFO HandlerThread:124 [system_monitor.py:probe():214] Collecting system info
2024-04-08 19:45:11,714 INFO SystemMonitor:124 [interfaces.py:start():190] Started cpu monitoring
2024-04-08 19:45:11,715 INFO SystemMonitor:124 [interfaces.py:start():190] Started disk monitoring
2024-04-08 19:45:11,716 INFO SystemMonitor:124 [interfaces.py:start():190] Started gpu monitoring
2024-04-08 19:45:11,717 INFO SystemMonitor:124 [interfaces.py:start():190] Started memory monitoring
2024-04-08 19:45:11,718 INFO SystemMonitor:124 [interfaces.py:start():190] Started network monitoring
2024-04-08 19:45:11,728 DEBUG HandlerThread:124 [system_info.py:probe():150] Probing system
2024-04-08 19:45:11,730 DEBUG HandlerThread:124 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-08 19:45:11,730 DEBUG HandlerThread:124 [system_info.py:probe():198] Probing system done
2024-04-08 19:45:11,730 DEBUG HandlerThread:124 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-08T19:45:11.728262', 'startedAt': '2024-04-08T19:45:11.064176', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'c6b69506ef79', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.124, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.124, 'min': 0.0, 'max': 0.0}, {'current': 2000.124, 'min': 0.0, 'max': 0.0}, {'current': 2000.124, 'min': 0.0, 'max': 0.0}, {'current': 2000.124, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5568.723930358887}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-08 19:45:11,730 INFO HandlerThread:124 [system_monitor.py:probe():224] Finished collecting system info
2024-04-08 19:45:11,730 INFO HandlerThread:124 [system_monitor.py:probe():227] Publishing system info
2024-04-08 19:45:11,730 DEBUG HandlerThread:124 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-08 19:45:12,610 INFO Thread-12 :124 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/conda-environment.yaml
2024-04-08 19:46:08,628 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/conda-environment.yaml
2024-04-08 19:46:09,053 DEBUG HandlerThread:124 [system_info.py:_save_conda():219] Saving conda packages done
2024-04-08 19:46:09,053 INFO HandlerThread:124 [system_monitor.py:probe():229] Finished publishing system info
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,063 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,064 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,064 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,064 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,064 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: python_packages
2024-04-08 19:46:09,064 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: stop_status
2024-04-08 19:46:09,064 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:09,065 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: pause
2024-04-08 19:46:09,065 INFO HandlerThread:124 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-04-08 19:46:09,065 INFO HandlerThread:124 [system_monitor.py:finish():203] Stopping system monitor
2024-04-08 19:46:09,066 INFO HandlerThread:124 [interfaces.py:finish():202] Joined cpu monitor
2024-04-08 19:46:09,068 INFO HandlerThread:124 [interfaces.py:finish():202] Joined disk monitor
2024-04-08 19:46:09,068 DEBUG SystemMonitor:124 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-08 19:46:09,068 DEBUG SystemMonitor:124 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-04-08 19:46:09,068 DEBUG SystemMonitor:124 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-04-08 19:46:09,069 DEBUG SenderThread:124 [sender.py:send_request():405] send_request: python_packages
2024-04-08 19:46:09,071 DEBUG SenderThread:124 [sender.py:send():378] send: telemetry
2024-04-08 19:46:09,071 DEBUG SenderThread:124 [sender.py:send_request():405] send_request: stop_status
2024-04-08 19:46:09,086 INFO HandlerThread:124 [interfaces.py:finish():202] Joined gpu monitor
2024-04-08 19:46:09,086 INFO HandlerThread:124 [interfaces.py:finish():202] Joined memory monitor
2024-04-08 19:46:09,086 INFO HandlerThread:124 [interfaces.py:finish():202] Joined network monitor
2024-04-08 19:46:09,087 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:09,189 DEBUG SenderThread:124 [sender.py:send():378] send: config
2024-04-08 19:46:09,190 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:09,190 DEBUG SenderThread:124 [sender.py:send():378] send: telemetry
2024-04-08 19:46:09,190 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:09,190 WARNING SenderThread:124 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-04-08 19:46:09,190 DEBUG SenderThread:124 [sender.py:send():378] send: telemetry
2024-04-08 19:46:09,191 DEBUG SenderThread:124 [sender.py:send():378] send: files
2024-04-08 19:46:09,191 INFO SenderThread:124 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now
2024-04-08 19:46:09,192 DEBUG SenderThread:124 [sender.py:send():378] send: stats
2024-04-08 19:46:09,424 INFO wandb-upload_0:124 [upload_job.py:push():131] Uploaded file /tmp/tmptbku1344wandb/6rd0hhnc-wandb-metadata.json
2024-04-08 19:46:09,628 INFO Thread-12 :124 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/output.log
2024-04-08 19:46:09,628 INFO Thread-12 :124 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/requirements.txt
2024-04-08 19:46:09,629 INFO Thread-12 :124 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/wandb-metadata.json
2024-04-08 19:46:11,629 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/output.log
2024-04-08 19:46:12,198 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:12,629 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/config.yaml
2024-04-08 19:46:13,742 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:17,279 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:18,743 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:21,596 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: resume
2024-04-08 19:46:21,596 INFO HandlerThread:124 [handler.py:handle_request_resume():699] starting system metrics thread
2024-04-08 19:46:21,596 INFO HandlerThread:124 [system_monitor.py:start():194] Starting system monitor
2024-04-08 19:46:21,596 INFO SystemMonitor:124 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-08 19:46:21,597 INFO SystemMonitor:124 [interfaces.py:start():190] Started cpu monitoring
2024-04-08 19:46:21,598 INFO SystemMonitor:124 [interfaces.py:start():190] Started disk monitoring
2024-04-08 19:46:21,599 INFO SystemMonitor:124 [interfaces.py:start():190] Started gpu monitoring
2024-04-08 19:46:21,601 INFO SystemMonitor:124 [interfaces.py:start():190] Started memory monitoring
2024-04-08 19:46:21,602 INFO SystemMonitor:124 [interfaces.py:start():190] Started network monitoring
2024-04-08 19:46:22,281 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:23,744 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:25,657 DEBUG SenderThread:124 [sender.py:send():378] send: config
2024-04-08 19:46:25,659 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:25,659 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:25,659 WARNING SenderThread:124 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-04-08 19:46:27,634 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/output.log
2024-04-08 19:46:27,660 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:28,763 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:32,660 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:33,769 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:37,661 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:38,780 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:42,667 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:43,639 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/config.yaml
2024-04-08 19:46:43,787 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:47,855 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:48,793 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:52,855 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:53,794 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:57,856 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:46:57,873 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: partial_history
2024-04-08 19:46:57,875 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:57,875 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:57,875 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:57,875 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:46:57,876 DEBUG SenderThread:124 [sender.py:send():378] send: history
2024-04-08 19:46:57,876 DEBUG SenderThread:124 [sender.py:send_request():405] send_request: summary_record
2024-04-08 19:46:57,877 INFO SenderThread:124 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-04-08 19:46:58,645 INFO Thread-12 :124 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/wandb-summary.json
2024-04-08 19:46:58,795 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:46:59,645 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/output.log
2024-04-08 19:47:03,711 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:03,796 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:08,712 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:09,125 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:13,718 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:14,131 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:14,650 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/config.yaml
2024-04-08 19:47:18,821 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:19,138 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:21,602 DEBUG SystemMonitor:124 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-08 19:47:21,604 DEBUG SenderThread:124 [sender.py:send():378] send: stats
2024-04-08 19:47:24,139 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:24,605 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:27,343 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: partial_history
2024-04-08 19:47:27,345 DEBUG SenderThread:124 [sender.py:send():378] send: history
2024-04-08 19:47:27,345 DEBUG SenderThread:124 [sender.py:send_request():405] send_request: summary_record
2024-04-08 19:47:27,345 INFO SenderThread:124 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-04-08 19:47:27,655 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/wandb-summary.json
2024-04-08 19:47:29,145 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:29,655 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/output.log
2024-04-08 19:47:30,216 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:34,151 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:35,216 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:39,152 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: keepalive
2024-04-08 19:47:40,217 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: status_report
2024-04-08 19:47:41,084 DEBUG HandlerThread:124 [handler.py:handle_request():146] handle_request: partial_history
2024-04-08 19:47:41,086 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:47:41,086 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:47:41,086 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:47:41,086 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:47:41,086 DEBUG SenderThread:124 [sender.py:send():378] send: metric
2024-04-08 19:47:41,086 DEBUG SenderThread:124 [sender.py:send():378] send: history
2024-04-08 19:47:41,087 DEBUG SenderThread:124 [sender.py:send_request():405] send_request: summary_record
2024-04-08 19:47:41,087 INFO SenderThread:124 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-04-08 19:47:41,659 INFO Thread-12 :124 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_194511-wp046ftq/files/wandb-summary.json
|