| 2024-02-28 16:47:28,218 INFO StreamThr :175 [internal.py:wandb_internal():86] W&B internal server running at pid: 175, started at: 2024-02-28 16:47:28.218262 | |
| 2024-02-28 16:47:28,221 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-28 16:47:28,517 INFO WriterThread:175 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240228_164728-a7eaa078/run-a7eaa078.wandb | |
| 2024-02-28 16:47:28,518 DEBUG SenderThread:175 [sender.py:send():382] send: header | |
| 2024-02-28 16:47:28,520 DEBUG SenderThread:175 [sender.py:send():382] send: run | |
| 2024-02-28 16:47:28,717 INFO SenderThread:175 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files | |
| 2024-02-28 16:47:28,718 INFO SenderThread:175 [sender.py:_start_run_threads():1136] run started: a7eaa078 with start time 1709138848.222988 | |
| 2024-02-28 16:47:28,726 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-28 16:47:28,726 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-28 16:47:28,811 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-28 16:47:28,822 DEBUG HandlerThread:175 [system_info.py:__init__():27] System info init | |
| 2024-02-28 16:47:28,822 DEBUG HandlerThread:175 [system_info.py:__init__():42] System info init done | |
| 2024-02-28 16:47:28,822 INFO HandlerThread:175 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-28 16:47:28,823 INFO SystemMonitor:175 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-28 16:47:28,823 INFO HandlerThread:175 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-28 16:47:28,823 INFO SystemMonitor:175 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-28 16:47:28,824 INFO SystemMonitor:175 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-28 16:47:28,825 INFO SystemMonitor:175 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-28 16:47:28,825 INFO SystemMonitor:175 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-28 16:47:28,828 INFO SystemMonitor:175 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-28 16:47:28,837 DEBUG HandlerThread:175 [system_info.py:probe():151] Probing system | |
| 2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [system_info.py:probe():199] Probing system done | |
| 2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-02-28T16:47:28.837546', 'startedAt': '2024-02-28T16:47:28.210756', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'bafa38fbdbe9', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.198, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}, {'current': 2000.198, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5520.956398010254}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-02-28 16:47:28,839 INFO HandlerThread:175 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-28 16:47:28,839 INFO HandlerThread:175 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-28 16:47:28,839 DEBUG HandlerThread:175 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment | |
| 2024-02-28 16:47:29,719 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/conda-environment.yaml | |
| 2024-02-28 16:48:23,617 DEBUG HandlerThread:175 [system_info.py:_save_conda():220] Saving conda packages done | |
| 2024-02-28 16:48:23,618 INFO HandlerThread:175 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-28 16:48:23,632 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,632 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,633 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-28 16:48:23,634 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-28 16:48:23,635 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-28 16:48:23,635 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,635 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-28 16:48:23,635 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,637 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,637 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:23,638 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:23,638 DEBUG SenderThread:175 [sender.py:send():382] send: telemetry | |
| 2024-02-28 16:48:23,639 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-28 16:48:23,695 DEBUG SenderThread:175 [sender.py:send():382] send: config | |
| 2024-02-28 16:48:23,696 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: telemetry | |
| 2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:48:23,697 WARNING SenderThread:175 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: telemetry | |
| 2024-02-28 16:48:23,697 DEBUG SenderThread:175 [sender.py:send():382] send: files | |
| 2024-02-28 16:48:23,697 INFO SenderThread:175 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-28 16:48:23,734 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/conda-environment.yaml | |
| 2024-02-28 16:48:23,734 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-metadata.json | |
| 2024-02-28 16:48:23,734 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/requirements.txt | |
| 2024-02-28 16:48:23,939 INFO wandb-upload_0:175 [upload_job.py:push():131] Uploaded file /tmp/tmpfgry5sqywandb/n1d5pray-wandb-metadata.json | |
| 2024-02-28 16:48:24,698 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:24,856 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:28,828 DEBUG SystemMonitor:175 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-28 16:48:28,830 DEBUG SenderThread:175 [sender.py:send():382] send: stats | |
| 2024-02-28 16:48:29,836 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:29,857 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:30,736 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/config.yaml | |
| 2024-02-28 16:48:34,858 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:34,966 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:39,859 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:39,967 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:44,859 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:44,968 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:49,860 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:49,968 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:54,861 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:48:54,969 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:48:58,831 DEBUG SenderThread:175 [sender.py:send():382] send: stats | |
| 2024-02-28 16:48:59,862 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:00,832 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:04,863 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:05,832 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:09,863 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:10,833 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:14,864 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:15,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:19,865 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:20,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:24,866 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:25,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:28,832 DEBUG SenderThread:175 [sender.py:send():382] send: stats | |
| 2024-02-28 16:49:29,867 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:31,833 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:34,868 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:36,833 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:39,875 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:41,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:44,876 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:46,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:49,877 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:51,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:54,878 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:49:56,836 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:49:58,833 DEBUG SenderThread:175 [sender.py:send():382] send: stats | |
| 2024-02-28 16:49:59,879 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:02,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:04,879 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:07,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:09,883 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:12,835 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:14,884 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:17,836 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:19,885 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:22,837 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:24,886 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:27,837 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:28,833 DEBUG SenderThread:175 [sender.py:send():382] send: stats | |
| 2024-02-28 16:50:29,914 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:33,834 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:35,220 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:37,773 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/output.log | |
| 2024-02-28 16:50:39,441 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:39,774 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/output.log | |
| 2024-02-28 16:50:40,252 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:44,442 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:45,208 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-28 16:50:45,211 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,211 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,212 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:45,213 DEBUG SenderThread:175 [sender.py:send():382] send: history | |
| 2024-02-28 16:50:45,213 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-28 16:50:45,213 INFO SenderThread:175 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-28 16:50:45,279 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-28 16:50:45,776 INFO Thread-12 :175 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-summary.json | |
| 2024-02-28 16:50:46,908 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: metric | |
| 2024-02-28 16:50:46,910 DEBUG SenderThread:175 [sender.py:send():382] send: history | |
| 2024-02-28 16:50:46,911 DEBUG SenderThread:175 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-28 16:50:46,911 INFO SenderThread:175 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-28 16:50:47,776 INFO Thread-12 :175 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240228_164728-a7eaa078/files/wandb-summary.json | |
| 2024-02-28 16:50:49,912 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-28 16:50:50,299 DEBUG HandlerThread:175 [handler.py:handle_request():146] handle_request: keepalive | |