2024-04-10 21:51:01,073 INFO    StreamThr :274 [internal.py:wandb_internal():86] W&B internal server running at pid: 274, started at: 2024-04-10 21:51:01.072932
2024-04-10 21:51:01,075 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status
2024-04-10 21:51:01,447 INFO    WriterThread:274 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240410_215101-ta1pahei/run-ta1pahei.wandb
2024-04-10 21:51:01,448 DEBUG   SenderThread:274 [sender.py:send():379] send: header
2024-04-10 21:51:01,459 DEBUG   SenderThread:274 [sender.py:send():379] send: run
2024-04-10 21:51:01,670 INFO    SenderThread:274 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files
2024-04-10 21:51:01,670 INFO    SenderThread:274 [sender.py:_start_run_threads():1124] run started: ta1pahei with start time 1712785861.074028
2024-04-10 21:51:01,678 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: check_version
2024-04-10 21:51:01,678 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: check_version
2024-04-10 21:51:01,777 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: run_start
2024-04-10 21:51:01,788 DEBUG   HandlerThread:274 [system_info.py:__init__():26] System info init
2024-04-10 21:51:01,789 DEBUG   HandlerThread:274 [system_info.py:__init__():41] System info init done
2024-04-10 21:51:01,789 INFO    HandlerThread:274 [system_monitor.py:start():194] Starting system monitor
2024-04-10 21:51:01,789 INFO    SystemMonitor:274 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-10 21:51:01,789 INFO    HandlerThread:274 [system_monitor.py:probe():214] Collecting system info
2024-04-10 21:51:01,789 INFO    SystemMonitor:274 [interfaces.py:start():190] Started cpu monitoring
2024-04-10 21:51:01,790 INFO    SystemMonitor:274 [interfaces.py:start():190] Started disk monitoring
2024-04-10 21:51:01,791 INFO    SystemMonitor:274 [interfaces.py:start():190] Started gpu monitoring
2024-04-10 21:51:01,792 INFO    SystemMonitor:274 [interfaces.py:start():190] Started memory monitoring
2024-04-10 21:51:01,792 INFO    SystemMonitor:274 [interfaces.py:start():190] Started network monitoring
2024-04-10 21:51:01,802 DEBUG   HandlerThread:274 [system_info.py:probe():150] Probing system
2024-04-10 21:51:01,805 DEBUG   HandlerThread:274 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-10 21:51:01,805 DEBUG   HandlerThread:274 [system_info.py:probe():198] Probing system done
2024-04-10 21:51:01,805 DEBUG   HandlerThread:274 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-10T21:51:01.802955', 'startedAt': '2024-04-10T21:51:01.066772', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '4bf7c58fba03', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.192, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.192, 'min': 0.0, 'max': 0.0}, {'current': 2000.192, 'min': 0.0, 'max': 0.0}, {'current': 2000.192, 'min': 0.0, 'max': 0.0}, {'current': 2000.192, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5578.723789215088}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-10 21:51:01,805 INFO    HandlerThread:274 [system_monitor.py:probe():224] Finished collecting system info
2024-04-10 21:51:01,805 INFO    HandlerThread:274 [system_monitor.py:probe():227] Publishing system info
2024-04-10 21:51:01,805 DEBUG   HandlerThread:274 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-10 21:51:02,672 INFO    Thread-12 :274 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/conda-environment.yaml
2024-04-10 21:51:16,821 ERROR   HandlerThread:274 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
    subprocess.call(
  File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
    return p.wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-10 21:51:16,824 DEBUG   HandlerThread:274 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-10 21:51:16,825 INFO    HandlerThread:274 [system_monitor.py:probe():229] Finished publishing system info
2024-04-10 21:51:16,831 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:16,832 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: keepalive
2024-04-10 21:51:16,832 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:16,832 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: keepalive
2024-04-10 21:51:16,832 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:16,832 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: keepalive
2024-04-10 21:51:16,833 DEBUG   SenderThread:274 [sender.py:send():379] send: files
2024-04-10 21:51:16,833 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-10 21:51:17,270 INFO    wandb-upload_0:274 [upload_job.py:push():131] Uploaded file /tmp/tmphet9wukmwandb/tfhoobcd-wandb-metadata.json
2024-04-10 21:51:17,675 INFO    Thread-12 :274 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/wandb-metadata.json
2024-04-10 21:51:17,776 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: python_packages
2024-04-10 21:51:17,776 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: python_packages
2024-04-10 21:51:17,779 DEBUG   SenderThread:274 [sender.py:send():379] send: telemetry
2024-04-10 21:51:17,789 DEBUG   SenderThread:274 [sender.py:send():379] send: config
2024-04-10 21:51:17,791 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 21:51:17,792 DEBUG   SenderThread:274 [sender.py:send():379] send: telemetry
2024-04-10 21:51:17,792 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 21:51:17,792 WARNING SenderThread:274 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-10 21:51:17,792 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:51:17,793 DEBUG   SenderThread:274 [sender.py:send():379] send: telemetry
2024-04-10 21:51:17,793 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:51:17,793 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:51:18,676 INFO    Thread-12 :274 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/requirements.txt
2024-04-10 21:51:18,676 INFO    Thread-12 :274 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 21:51:20,677 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 21:51:22,163 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:22,849 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: pause
2024-04-10 21:51:22,849 INFO    HandlerThread:274 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-04-10 21:51:22,849 INFO    HandlerThread:274 [system_monitor.py:finish():203] Stopping system monitor
2024-04-10 21:51:22,849 DEBUG   SystemMonitor:274 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-10 21:51:22,849 DEBUG   SystemMonitor:274 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-04-10 21:51:22,849 DEBUG   SystemMonitor:274 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-04-10 21:51:22,850 INFO    HandlerThread:274 [interfaces.py:finish():202] Joined cpu monitor
2024-04-10 21:51:22,850 INFO    HandlerThread:274 [interfaces.py:finish():202] Joined disk monitor
2024-04-10 21:51:22,861 INFO    HandlerThread:274 [interfaces.py:finish():202] Joined gpu monitor
2024-04-10 21:51:22,861 INFO    HandlerThread:274 [interfaces.py:finish():202] Joined memory monitor
2024-04-10 21:51:22,861 INFO    HandlerThread:274 [interfaces.py:finish():202] Joined network monitor
2024-04-10 21:51:22,861 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:51:27,862 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:32,778 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:51:32,779 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:51:32,779 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:51:33,855 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:34,682 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/config.yaml
2024-04-10 21:51:38,978 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:43,979 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:47,777 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:51:47,778 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:51:47,817 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:51:49,947 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:54,947 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:51:59,948 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:02,777 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:52:02,778 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:52:02,818 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:52:05,863 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:10,864 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:15,865 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:17,777 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:52:17,778 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:52:17,818 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:52:20,900 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:25,901 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:30,902 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:32,777 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:52:32,778 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:52:32,818 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:52:36,843 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:40,759 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: resume
2024-04-10 21:52:40,759 INFO    HandlerThread:274 [handler.py:handle_request_resume():699] starting system metrics thread
2024-04-10 21:52:40,759 INFO    HandlerThread:274 [system_monitor.py:start():194] Starting system monitor
2024-04-10 21:52:40,759 INFO    SystemMonitor:274 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-10 21:52:40,760 INFO    SystemMonitor:274 [interfaces.py:start():190] Started cpu monitoring
2024-04-10 21:52:40,761 INFO    SystemMonitor:274 [interfaces.py:start():190] Started disk monitoring
2024-04-10 21:52:40,762 INFO    SystemMonitor:274 [interfaces.py:start():190] Started gpu monitoring
2024-04-10 21:52:40,763 INFO    SystemMonitor:274 [interfaces.py:start():190] Started memory monitoring
2024-04-10 21:52:40,764 INFO    SystemMonitor:274 [interfaces.py:start():190] Started network monitoring
2024-04-10 21:52:41,844 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:46,708 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 21:52:47,665 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:47,789 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:52:47,789 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:52:47,829 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:52:48,709 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 21:52:52,915 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:52:57,916 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:02,783 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:53:02,784 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:53:02,784 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:53:04,789 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:09,789 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:14,790 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:17,800 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:53:17,802 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:53:17,802 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:53:19,871 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:24,874 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:29,875 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:32,801 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:53:32,801 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:53:32,802 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:53:34,952 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:39,953 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:40,767 DEBUG   SystemMonitor:274 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-10 21:53:40,768 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:53:45,770 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:48,058 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:53:48,058 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:53:48,106 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:53:51,112 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:53:56,113 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:01,113 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:03,034 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:54:03,035 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:54:03,075 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:54:06,135 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:10,769 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:54:11,770 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:16,771 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:18,045 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:54:18,045 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:54:18,087 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:54:22,200 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:27,201 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:32,202 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:33,045 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:54:33,045 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:54:33,085 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:54:37,214 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:40,770 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:54:42,771 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:47,772 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:48,045 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:54:48,046 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:54:48,087 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:54:53,167 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:54:58,167 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:03,168 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:04,522 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:55:04,523 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:55:04,524 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:55:08,693 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:10,771 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:55:13,772 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:18,288 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:55:18,288 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:55:18,399 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:55:19,387 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:24,388 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:29,389 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:33,895 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:55:33,895 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:55:33,896 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:55:34,984 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:39,985 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:40,772 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:55:45,774 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:48,531 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:55:48,581 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:55:48,582 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:55:50,895 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:55:55,897 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:00,897 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:03,531 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:56:03,581 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:56:03,582 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:56:06,895 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:10,774 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:56:12,775 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:17,775 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:18,531 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:56:18,581 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:56:18,582 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:56:22,895 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:27,896 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:32,898 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:33,722 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:56:33,729 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:56:33,729 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:56:38,159 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:40,775 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:56:43,776 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:48,546 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:56:48,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:56:48,586 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:56:49,717 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:56:55,577 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:00,578 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:03,546 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:57:03,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:57:03,586 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:57:05,636 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:10,637 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:10,776 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:57:15,777 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:18,547 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:57:18,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:57:18,586 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:57:21,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:26,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:31,587 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:33,567 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:57:33,588 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:57:33,589 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:57:36,605 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:40,777 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:57:41,778 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:46,779 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:48,567 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:57:48,588 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:57:48,589 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:57:52,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:57:57,587 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:02,588 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:03,567 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:58:03,588 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:58:03,589 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:58:08,587 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:10,778 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:58:13,779 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:18,567 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:58:18,588 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:58:18,589 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:58:19,703 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:25,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:30,587 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:33,567 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:58:33,589 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:58:33,589 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:58:36,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:40,779 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:58:41,780 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:46,780 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:48,567 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:58:48,602 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:58:48,602 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:58:52,596 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:58:57,597 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:02,598 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:03,568 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:59:03,589 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:59:03,589 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:59:08,596 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:10,780 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:59:13,781 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:18,568 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:59:18,589 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:59:18,590 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:59:19,663 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:25,597 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:30,598 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:33,568 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:59:33,589 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:59:33,590 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:59:35,711 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:40,781 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 21:59:40,783 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:45,783 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:48,568 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 21:59:48,589 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 21:59:48,590 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 21:59:51,602 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 21:59:56,604 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:01,605 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:03,568 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:00:03,589 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:00:03,590 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:00:06,608 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:10,782 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 22:00:11,783 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:16,784 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:18,568 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:00:18,590 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:00:18,590 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:00:22,603 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:27,604 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:32,605 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:33,584 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:00:33,614 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:00:33,615 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:00:38,603 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:40,783 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 22:00:44,504 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:44,558 DEBUG   SenderThread:274 [sender.py:send():379] send: config
2024-04-10 22:00:44,560 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 22:00:44,560 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 22:00:44,560 WARNING SenderThread:274 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-10 22:00:44,919 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:00:46,920 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:00:48,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:00:48,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:00:48,615 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:00:48,921 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:00:49,760 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:55,604 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:00:56,923 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:00:58,924 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:01:00,664 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:03,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:01:03,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:01:03,615 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:01:05,691 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:05,927 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/config.yaml
2024-04-10 22:01:08,928 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:01:10,784 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 22:01:11,785 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:16,786 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:18,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:01:18,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:01:18,616 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:01:22,656 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:27,657 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:32,658 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:33,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:01:33,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:01:33,615 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:01:38,656 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:40,785 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 22:01:43,786 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:48,588 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:01:48,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:01:48,615 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:01:49,779 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:54,904 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: partial_history
2024-04-10 22:01:54,907 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 22:01:54,907 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 22:01:54,907 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 22:01:54,907 DEBUG   SenderThread:274 [sender.py:send():379] send: metric
2024-04-10 22:01:54,907 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:01:54,907 DEBUG   SenderThread:274 [sender.py:send():379] send: history
2024-04-10 22:01:54,908 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:54,908 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:54,945 INFO    Thread-12 :274 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/wandb-summary.json
2024-04-10 22:01:56,946 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:01:57,338 DEBUG   SenderThread:274 [sender.py:send():379] send: telemetry
2024-04-10 22:01:57,338 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:57,340 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: partial_history
2024-04-10 22:01:57,341 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:57,342 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:57,342 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:57,342 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:57,343 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:57,343 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:57,343 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:57,344 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:57,344 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:57,344 DEBUG   SenderThread:274 [sender.py:send():379] send: history
2024-04-10 22:01:57,345 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: summary_record
2024-04-10 22:01:57,345 INFO    SenderThread:274 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-10 22:01:57,946 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/wandb-summary.json
2024-04-10 22:01:58,946 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/output.log
2024-04-10 22:02:00,489 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:02:03,585 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:02:03,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:02:03,616 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:02:05,656 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:02:10,662 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:02:10,786 DEBUG   SenderThread:274 [sender.py:send():379] send: stats
2024-04-10 22:02:10,950 INFO    Thread-12 :274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240410_215101-ta1pahei/files/config.yaml
2024-04-10 22:02:15,787 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:02:18,586 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-10 22:02:18,615 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: stop_status
2024-04-10 22:02:18,616 DEBUG   SenderThread:274 [sender.py:send_request():406] send_request: stop_status
2024-04-10 22:02:21,656 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report
2024-04-10 22:02:26,657 DEBUG   HandlerThread:274 [handler.py:handle_request():146] handle_request: status_report