2024-04-12 07:40:44,508 INFO    StreamThr :485 [internal.py:wandb_internal():86] W&B internal server running at pid: 485, started at: 2024-04-12 07:40:44.508065
2024-04-12 07:40:44,510 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status
2024-04-12 07:40:44,782 INFO    WriterThread:485 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240412_074044-widovcn3/run-widovcn3.wandb
2024-04-12 07:40:44,782 DEBUG   SenderThread:485 [sender.py:send():379] send: header
2024-04-12 07:40:44,785 DEBUG   SenderThread:485 [sender.py:send():379] send: run
2024-04-12 07:40:44,894 INFO    SenderThread:485 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240412_074044-widovcn3/files
2024-04-12 07:40:44,895 INFO    SenderThread:485 [sender.py:_start_run_threads():1124] run started: widovcn3 with start time 1712907644.507718
2024-04-12 07:40:44,903 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: check_version
2024-04-12 07:40:44,904 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: check_version
2024-04-12 07:40:44,996 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: run_start
2024-04-12 07:40:45,008 DEBUG   HandlerThread:485 [system_info.py:__init__():26] System info init
2024-04-12 07:40:45,008 DEBUG   HandlerThread:485 [system_info.py:__init__():41] System info init done
2024-04-12 07:40:45,008 INFO    HandlerThread:485 [system_monitor.py:start():194] Starting system monitor
2024-04-12 07:40:45,008 INFO    SystemMonitor:485 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-12 07:40:45,008 INFO    HandlerThread:485 [system_monitor.py:probe():214] Collecting system info
2024-04-12 07:40:45,009 INFO    SystemMonitor:485 [interfaces.py:start():190] Started cpu monitoring
2024-04-12 07:40:45,009 INFO    SystemMonitor:485 [interfaces.py:start():190] Started disk monitoring
2024-04-12 07:40:45,011 INFO    SystemMonitor:485 [interfaces.py:start():190] Started gpu monitoring
2024-04-12 07:40:45,012 INFO    SystemMonitor:485 [interfaces.py:start():190] Started memory monitoring
2024-04-12 07:40:45,013 INFO    SystemMonitor:485 [interfaces.py:start():190] Started network monitoring
2024-04-12 07:40:45,023 DEBUG   HandlerThread:485 [system_info.py:probe():150] Probing system
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [gitlib.py:_init_repo():56] git repository is invalid
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [system_info.py:probe():198] Probing system done
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-12T07:40:45.023659', 'startedAt': '2024-04-12T07:40:44.501264', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'e5a48bec8248', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.138, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}, {'current': 2000.138, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5565.782573699951}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}}
2024-04-12 07:40:45,026 INFO    HandlerThread:485 [system_monitor.py:probe():224] Finished collecting system info
2024-04-12 07:40:45,026 INFO    HandlerThread:485 [system_monitor.py:probe():227] Publishing system info
2024-04-12 07:40:45,026 DEBUG   HandlerThread:485 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-12 07:40:45,897 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/conda-environment.yaml
2024-04-12 07:41:00,040 ERROR   HandlerThread:485 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
    subprocess.call(
  File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
    return p.wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-04-12 07:41:00,041 DEBUG   HandlerThread:485 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-12 07:41:00,042 INFO    HandlerThread:485 [system_monitor.py:probe():229] Finished publishing system info
2024-04-12 07:41:00,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:00,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:41:00,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:00,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:41:00,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:00,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: keepalive
2024-04-12 07:41:00,048 DEBUG   SenderThread:485 [sender.py:send():379] send: files
2024-04-12 07:41:00,049 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-12 07:41:00,245 INFO    wandb-upload_0:485 [upload_job.py:push():131] Uploaded file /tmp/tmpqpxjmay6wandb/iki4gmew-wandb-metadata.json
2024-04-12 07:41:00,900 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-metadata.json
2024-04-12 07:41:01,034 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: python_packages
2024-04-12 07:41:01,034 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: python_packages
2024-04-12 07:41:01,038 DEBUG   SenderThread:485 [sender.py:send():379] send: telemetry
2024-04-12 07:41:01,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:01,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:01,102 DEBUG   SenderThread:485 [sender.py:send():379] send: config
2024-04-12 07:41:01,103 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:01,103 DEBUG   SenderThread:485 [sender.py:send():379] send: telemetry
2024-04-12 07:41:01,104 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:01,104 WARNING SenderThread:485 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-12 07:41:01,104 DEBUG   SenderThread:485 [sender.py:send():379] send: telemetry
2024-04-12 07:41:01,900 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:01,901 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/requirements.txt
2024-04-12 07:41:03,901 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:05,568 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:10,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:15,575 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:15,907 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/config.yaml
2024-04-12 07:41:16,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:16,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:16,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:21,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:22,063 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: metric
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:41:22,066 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:41:22,068 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:41:22,910 INFO    Thread-12 :485 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:41:23,910 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:26,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:31,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:31,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:31,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:32,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:37,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:42,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:45,014 DEBUG   SystemMonitor:485 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-12 07:41:45,015 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:41:46,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:41:46,035 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:41:46,076 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:41:46,285 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:41:46,286 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:41:46,287 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:41:46,287 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:41:46,921 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:41:47,922 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:41:48,004 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:48,922 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/config.yaml
2024-04-12 07:41:53,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:41:58,107 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:01,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:01,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:01,038 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:03,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:08,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:09,300 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:42:09,301 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:42:09,301 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:42:09,303 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:42:09,932 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:42:11,933 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:42:14,018 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:15,016 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:42:16,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:16,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:16,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:19,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:24,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:29,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:31,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:31,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:31,076 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:34,125 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:35,366 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:42:35,367 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:42:35,367 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:42:35,369 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:42:35,942 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:42:37,943 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:42:40,060 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:45,017 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:42:46,018 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:46,035 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:42:46,036 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:42:46,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:42:51,081 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:42:56,082 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:00,593 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:43:00,594 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:43:00,594 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:43:00,595 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:43:00,953 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:43:01,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:01,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:01,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:01,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:03,954 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:43:06,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:11,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:15,018 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:43:16,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:16,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:16,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:17,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:22,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:25,211 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:43:25,212 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:43:25,213 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:43:25,213 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:43:25,965 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:43:27,934 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:27,966 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:43:31,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:31,037 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:31,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:33,175 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:38,176 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:43,177 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:45,019 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:43:46,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:43:46,037 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:43:46,077 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:43:47,760 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:43:47,761 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:43:47,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:43:47,763 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:43:47,973 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:43:48,211 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:49,974 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:43:53,212 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:43:58,213 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:01,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:01,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:04,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:09,069 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:14,070 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:15,021 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:44:16,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:16,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:16,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:16,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:44:16,359 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:44:16,359 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:44:16,360 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:44:16,986 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:44:17,987 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:44:19,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:24,085 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:29,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:31,042 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:31,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:34,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:39,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:39,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:44:39,437 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:44:39,437 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:44:39,439 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:44:39,995 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:44:41,996 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:44:44,242 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:45,023 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:44:46,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:44:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:44:46,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:44:50,094 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:44:55,095 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:00,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:01,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:01,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:01,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:03,599 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:45:03,600 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:45:03,600 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:45:03,601 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:45:04,005 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:45:05,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:06,006 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:45:10,364 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:15,024 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:45:16,025 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:16,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:16,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:21,075 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:24,892 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:45:24,894 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:45:24,894 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:45:24,896 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:45:25,016 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:45:26,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:28,017 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:45:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:31,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:32,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:37,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:42,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:45,025 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:45:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:45:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:45:46,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:45:47,112 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:48,638 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:45:48,639 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:45:48,639 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:45:48,640 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:45:49,027 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:45:50,028 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:45:52,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:45:57,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:01,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:01,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:01,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:03,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:08,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:13,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:15,026 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:46:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:16,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:16,643 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:46:16,644 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:46:16,644 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:46:16,645 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:46:17,039 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:46:18,040 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:46:18,335 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:23,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:28,337 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:31,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:31,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:31,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:34,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:39,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:44,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:45,027 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:46:46,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:46:46,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:46:46,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:46:46,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:46:46,159 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:46:46,159 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:46:46,160 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:46:47,054 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:46:48,054 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:46:49,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:54,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:46:59,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:01,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:01,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:01,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:05,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:10,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:11,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:47:11,147 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:47:11,147 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:47:11,147 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:47:12,063 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:47:14,064 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:47:15,028 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:47:16,028 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:16,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:16,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:16,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:21,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:26,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:31,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:31,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:31,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:32,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:35,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:47:35,138 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:47:35,138 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:47:35,139 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:47:36,075 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:47:37,863 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:38,076 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:47:42,864 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:45,029 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:47:46,042 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:47:46,044 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:47:46,044 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:47:48,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:53,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:47:58,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:00,044 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:48:00,045 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:48:00,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:48:00,046 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:48:00,086 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:48:01,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:01,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:01,051 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:02,087 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:48:04,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:09,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:14,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:15,034 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:48:16,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:16,045 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:16,085 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:20,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:25,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:25,939 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:48:25,941 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:48:25,941 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:48:25,941 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:48:26,098 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:48:28,098 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:48:30,668 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:31,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:31,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:31,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:36,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:41,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:45,031 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:48:46,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:48:46,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:48:46,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:48:46,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:49,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:48:49,772 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:48:49,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:48:49,774 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:48:50,107 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:48:51,487 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:48:53,109 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:48:56,488 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:01,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:01,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:01,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:02,083 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:07,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:12,085 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:15,032 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:49:15,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:49:15,267 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:49:15,267 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:49:15,268 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:49:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:16,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:16,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:16,119 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:49:17,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:19,120 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:49:22,090 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:27,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:31,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:32,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:37,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:38,408 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:49:38,409 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:49:38,409 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:49:38,412 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:49:39,128 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:49:41,129 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:49:42,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:45,033 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:49:46,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:49:46,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:49:46,050 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:49:48,164 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:53,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:49:58,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:01,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:01,076 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:02,019 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:50:02,021 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:50:02,021 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:50:02,021 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:50:02,136 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:50:03,788 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:05,137 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:50:08,789 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:13,790 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:15,034 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:50:16,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:16,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:19,090 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:23,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:50:23,389 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:50:23,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:50:23,390 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:50:24,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:24,184 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:50:27,186 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:50:29,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:31,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:35,072 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:40,073 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:45,035 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:50:46,036 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:46,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:50:46,046 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:50:46,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:50:50,249 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:50:50,250 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:50:50,251 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:50:50,251 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:50:51,194 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:50:51,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:50:53,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:50:56,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:01,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:01,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:01,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:02,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:07,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:12,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:15,036 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:51:15,993 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:51:15,994 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:51:15,995 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:51:15,995 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:51:16,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:16,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:16,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:16,205 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:51:17,738 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:19,207 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:51:22,739 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:27,740 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:31,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:31,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:33,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:38,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:43,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:43,648 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:51:43,649 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:51:43,649 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:51:43,649 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:51:44,219 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:51:45,038 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:51:46,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:51:46,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:51:46,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:51:47,220 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:51:49,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:54,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:51:59,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:01,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:01,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:05,109 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:09,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:52:09,163 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:52:09,163 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:52:09,163 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:52:09,228 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:52:10,896 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:11,229 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:52:15,039 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:52:16,040 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:16,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:16,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:21,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:26,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:31,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:31,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:31,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:32,074 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:34,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:52:34,132 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:52:34,132 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:52:34,132 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:52:34,237 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:52:37,238 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:52:37,867 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:42,868 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:45,040 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:52:46,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:52:46,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:52:46,050 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:52:48,079 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:53,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:52:57,000 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:52:57,001 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:52:57,002 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:52:57,002 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:52:57,246 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:52:58,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:01,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:01,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:01,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:01,247 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:53:04,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:09,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:14,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:15,041 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:53:16,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:16,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:16,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:20,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:20,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:53:20,633 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:53:20,633 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:53:20,633 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:53:21,254 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:53:23,255 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:53:25,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:30,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:31,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:31,050 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:31,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:36,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:41,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:43,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:53:43,277 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:53:43,277 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:53:43,277 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:53:44,262 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:53:45,042 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:53:46,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:53:46,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:53:46,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:53:47,104 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:47,264 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:53:52,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:53:57,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:01,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:01,048 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:01,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:02,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:07,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:07,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:54:07,632 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:54:07,633 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:54:07,633 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:54:08,272 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:54:11,274 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:54:12,325 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:15,044 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:54:16,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:16,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:16,049 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:18,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:23,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:28,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:28,915 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:54:28,916 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:54:28,917 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:54:28,917 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:54:29,280 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:54:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:31,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:31,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:31,281 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:54:34,099 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:39,100 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:44,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:45,045 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:54:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:54:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:54:46,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:54:49,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:54,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:55,471 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:54:55,473 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:54:55,473 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:54:55,474 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:54:56,290 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:54:59,163 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:54:59,292 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:55:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:01,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:01,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:05,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:10,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:15,047 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:55:16,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:16,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:16,051 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:16,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:21,039 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:55:21,041 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:55:21,041 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:55:21,041 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:55:21,301 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:55:21,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:23,302 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:55:26,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:31,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:32,156 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:37,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:42,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:44,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:55:44,186 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:55:44,187 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:55:44,187 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:55:44,311 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:55:45,047 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:55:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:55:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:55:46,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:55:47,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:47,312 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:55:52,186 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:55:57,187 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:01,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:03,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:08,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:11,587 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:56:11,588 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:56:11,589 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:56:11,589 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:56:12,321 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:56:13,299 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:15,049 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:56:15,322 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:56:16,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:16,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:16,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:19,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:24,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:29,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:31,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:33,453 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:56:33,454 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:56:33,455 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:56:33,455 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:56:34,183 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:34,330 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:56:35,330 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:56:39,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:44,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:45,050 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:56:46,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:56:46,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:56:46,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:56:50,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:54,460 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:56:54,461 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:56:54,461 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:56:54,462 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:56:55,183 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:56:55,339 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:56:57,340 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:57:00,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:01,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:01,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:01,052 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:06,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:11,081 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:15,054 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:57:16,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:16,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:16,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:17,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:19,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:57:19,401 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:57:19,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:57:19,402 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:57:20,350 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:57:21,350 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:57:22,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:27,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:31,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:31,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:31,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:33,082 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:38,083 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:43,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:43,651 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:57:43,652 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:57:43,652 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:57:43,653 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:57:44,359 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:57:45,052 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:57:46,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:57:46,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:57:46,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:57:47,361 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:57:48,095 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:53,096 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:57:58,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:01,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:01,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:01,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:03,104 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:06,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:58:06,144 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:58:06,144 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:58:06,145 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:58:06,368 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:58:08,860 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:09,370 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:58:13,861 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:15,053 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:58:16,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:16,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:16,053 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:19,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:24,087 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:29,088 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:31,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:31,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:31,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:31,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:58:31,421 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:58:31,422 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:58:31,424 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:58:32,378 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:58:33,379 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:58:34,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:39,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:44,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:45,054 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:58:46,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:58:46,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:58:46,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:58:50,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:55,013 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:58:55,014 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:58:55,014 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:58:55,015 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:58:55,388 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:58:55,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:58:57,388 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:59:00,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:01,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:01,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:01,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:06,180 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:11,181 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:15,056 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:59:16,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:16,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:16,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:17,156 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:19,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:59:19,268 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:59:19,268 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:59:19,269 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:59:19,398 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:59:21,398 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:59:22,993 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:27,995 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:31,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:31,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:31,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:33,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:38,154 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:43,155 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:45,057 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 07:59:45,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 07:59:45,602 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 07:59:45,603 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 07:59:45,603 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 07:59:46,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 07:59:46,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 07:59:46,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 07:59:46,409 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 07:59:47,409 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 07:59:48,396 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:53,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 07:59:58,398 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:01,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:01,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:01,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:04,365 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:09,366 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:11,984 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:00:11,985 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:00:11,985 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:00:11,986 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:00:12,418 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:00:14,698 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:15,058 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:00:15,419 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:00:16,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:16,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:16,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:20,422 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:25,423 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:30,424 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:31,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:31,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:31,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:36,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:00:36,161 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:00:36,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:00:36,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:36,162 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:00:36,427 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:00:39,428 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:00:42,012 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:45,059 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:00:46,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:00:46,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:00:46,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:00:47,379 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:52,380 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:00:57,381 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:01,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:01,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:01,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:02,667 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:01:02,668 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:01:02,668 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:01:02,668 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:02,669 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:01:03,437 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:01:05,439 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:01:08,417 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:13,418 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:15,060 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:01:16,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:16,287 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:16,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:19,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:24,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:25,810 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:01:25,811 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:01:25,812 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:01:25,812 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:01:26,449 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:01:27,449 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:01:29,572 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:31,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:31,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:31,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:35,396 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:40,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:45,061 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:01:46,062 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:46,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:01:46,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:01:46,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:01:51,408 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:01:52,071 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:01:52,072 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:01:52,073 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:01:52,073 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:01:52,460 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:01:55,461 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:01:56,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:01,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:01,290 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:01,290 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:02,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:07,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:12,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:15,062 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:02:16,240 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:02:16,241 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:02:16,242 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:02:16,242 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:02:16,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:16,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:16,290 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:16,469 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:02:17,979 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:19,471 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:02:22,980 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:27,981 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:31,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:31,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:31,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:33,409 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:38,410 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:39,529 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:02:39,531 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:02:39,531 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:02:39,531 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:02:40,479 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:02:41,479 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:02:44,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:45,064 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:02:46,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:02:46,289 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:02:46,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:02:49,362 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:54,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:02:59,364 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:01,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:01,289 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:01,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:05,330 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:07,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:03:07,185 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:03:07,186 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:03:07,186 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:03:07,490 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:03:09,491 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:03:10,947 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:15,065 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:03:16,066 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:16,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:16,288 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:16,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:21,356 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:26,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:30,601 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:03:30,602 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:03:30,602 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:03:30,603 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:03:31,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:31,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:31,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:31,385 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:31,501 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:03:33,502 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:03:36,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:41,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:45,066 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:03:46,307 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:03:46,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:03:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:03:47,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:51,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:03:51,372 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:03:51,372 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:03:51,372 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:03:51,510 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:03:53,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:03:53,510 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:03:58,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:01,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:01,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:01,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:03,360 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:08,361 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:13,362 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:15,067 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:04:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:16,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:16,350 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:16,758 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:04:16,760 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:04:16,760 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:04:16,761 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:04:17,520 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:04:18,510 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:19,521 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:04:23,511 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:28,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:31,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:31,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:34,356 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:39,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:42,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:04:42,656 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:04:42,656 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:04:42,656 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:04:43,531 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:04:44,365 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:45,069 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:04:45,532 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:04:46,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:04:46,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:04:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:04:50,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:04:55,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:00,359 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:01,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:01,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:01,349 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:05,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:10,421 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:10,582 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:05:10,583 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:05:10,583 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:05:10,584 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:05:11,543 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:05:13,544 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:05:15,070 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:05:16,071 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:16,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:16,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:21,394 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:26,395 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:31,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:31,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:31,335 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:32,335 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:32,484 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:05:32,485 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:05:32,486 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:05:32,486 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:05:32,553 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:05:35,554 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:05:37,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:42,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:45,071 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:05:46,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:05:46,309 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:05:46,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:05:47,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:52,421 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:05:55,258 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:05:55,260 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:05:55,260 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:05:55,261 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:05:55,562 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:05:57,563 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:05:58,003 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:01,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:01,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:01,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:03,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:08,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:13,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:15,077 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:06:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:16,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:16,350 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:18,427 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:20,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:06:20,570 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:06:20,571 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:06:20,571 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:06:20,574 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:06:23,576 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:06:24,347 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:29,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:31,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:31,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:31,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:34,356 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:39,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:44,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:45,073 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:06:46,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:06:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:06:46,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:06:47,067 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:06:47,069 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:06:47,069 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:06:47,071 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:06:47,586 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:06:49,586 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:06:49,816 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:54,817 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:06:59,818 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:01,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:01,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:01,350 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:05,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:10,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:12,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:07:12,555 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:07:12,556 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:07:12,556 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:07:12,596 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:07:15,075 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:07:15,597 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:07:16,075 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:16,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:16,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:21,416 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:26,417 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:31,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:31,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:31,351 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:32,393 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:37,394 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:37,530 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:07:37,531 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:07:37,531 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:07:37,534 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:07:37,605 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:07:39,606 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:07:43,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:45,076 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:07:46,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:07:46,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:07:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:07:48,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:53,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:07:58,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:01,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:01,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:01,344 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:03,838 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:08:03,839 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:08:03,840 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:08:03,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:03,840 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:08:04,615 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:08:05,616 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:08:09,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:14,550 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:15,077 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:08:16,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:16,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:16,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:20,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:25,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:26,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:08:26,841 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:08:26,842 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:08:26,842 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:08:27,624 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:08:29,625 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:08:30,589 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:31,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:31,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:31,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:36,367 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:41,368 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:45,078 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:08:46,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:08:46,311 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:08:46,343 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:08:47,343 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:52,339 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:08:52,340 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:08:52,340 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:08:52,341 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:08:52,633 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:08:53,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:08:53,634 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:08:58,048 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:01,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:01,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:01,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:03,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:08,442 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:13,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:15,080 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:09:16,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:16,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:16,352 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:16,855 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:09:16,856 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:09:16,856 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:09:16,857 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:09:17,643 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:09:18,845 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:19,644 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:09:23,846 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:28,847 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:31,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:31,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:34,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:37,604 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:09:37,606 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:09:37,606 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:09:37,606 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:09:37,652 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:09:39,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:39,653 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:09:44,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:45,081 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:09:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:09:46,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:09:46,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:09:49,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:54,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:09:58,710 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:09:58,711 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:09:58,712 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:09:58,712 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:09:59,661 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:10:00,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:01,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:01,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:01,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:01,662 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:10:06,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:11,371 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:15,081 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:10:16,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:16,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:16,352 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:16,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:21,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:22,705 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:10:22,706 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:10:22,707 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:10:22,707 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:10:23,670 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:10:25,671 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:10:27,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:31,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:31,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:33,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:38,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:43,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:45,083 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:10:46,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:10:46,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:10:46,353 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:10:48,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:49,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:10:49,688 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:10:49,688 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:10:49,689 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:10:50,682 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:10:51,682 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:10:54,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:10:59,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:01,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:01,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:01,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:04,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:09,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:14,442 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:15,083 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:11:16,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:16,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:16,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:16,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:11:16,774 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:11:16,775 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:11:16,775 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:11:17,694 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:11:19,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:19,695 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:11:24,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:29,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:31,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:31,316 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:31,316 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:35,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:40,210 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:11:40,211 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:11:40,211 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:11:40,212 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:11:40,704 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:11:40,933 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:41,705 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:11:45,085 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:11:46,086 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:11:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:11:46,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:11:51,409 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:11:56,410 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:01,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:01,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:01,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:02,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:04,914 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:12:04,915 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:12:04,915 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:12:04,916 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:12:05,715 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:12:07,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:07,716 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:12:12,655 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:15,086 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:12:16,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:16,313 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:16,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:18,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:23,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:28,437 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:31,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:31,374 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:12:31,375 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:12:31,376 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:12:31,376 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:12:31,726 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:12:33,729 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:12:34,257 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:39,258 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:44,259 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:45,087 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:12:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:12:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:12:46,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:12:49,392 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:54,393 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:12:55,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:12:55,099 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:12:55,099 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:12:55,099 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:12:55,738 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:12:58,739 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:12:59,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:01,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:01,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:05,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:10,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:15,088 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:13:16,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:16,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:16,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:16,355 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:19,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:13:19,508 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:13:19,509 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:13:19,509 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:13:19,748 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:13:21,253 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:22,749 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:13:26,254 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:31,255 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:31,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:31,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:31,314 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:36,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:41,458 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:45,089 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:13:45,227 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:13:45,228 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:13:45,229 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:13:45,229 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:13:45,759 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:13:46,313 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:13:46,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:13:46,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:13:47,341 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:48,760 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:13:52,342 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:13:57,343 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:01,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:03,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:08,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:09,458 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:14:09,459 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:14:09,459 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:14:09,459 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:14:09,768 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:14:12,770 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:14:14,192 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:15,090 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:14:16,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:16,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:16,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:19,369 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:24,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:29,371 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:31,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:31,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:31,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:33,527 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:14:33,528 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:14:33,528 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:14:33,528 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:14:33,778 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:14:35,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:36,780 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:14:40,276 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:45,091 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:14:46,092 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:46,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:14:46,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:14:46,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:14:51,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:54,679 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:14:54,680 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:14:54,680 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:14:54,680 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:14:54,786 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:14:56,399 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:14:56,787 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:15:01,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:01,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:01,316 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:01,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:06,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:11,437 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:15,097 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:15:15,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:15:15,808 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:15:15,808 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:15:15,808 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:15:16,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:16,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:16,504 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:16,544 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:16,795 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:15:18,796 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:15:21,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:26,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:31,497 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:31,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:31,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:32,576 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:37,577 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:39,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:15:39,619 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:15:39,620 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:15:39,620 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:15:39,803 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:15:42,805 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:15:43,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:45,094 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:15:46,497 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:15:46,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:15:46,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:15:48,638 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:53,639 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:15:58,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:00,473 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:16:00,475 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:16:00,475 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:16:00,477 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:16:00,812 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:16:01,498 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:01,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:01,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:02,812 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:16:04,621 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:09,622 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:14,622 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:15,095 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:16:16,498 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:16,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:16,539 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:20,573 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:22,223 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:16:22,224 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:16:22,224 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:16:22,225 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:16:22,820 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:16:24,821 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:16:25,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:30,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:31,498 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:31,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:31,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:36,584 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:41,585 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:45,095 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:16:46,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:16:46,503 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:16:46,543 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:16:46,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:47,880 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:16:47,881 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:16:47,881 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:16:47,882 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:16:48,830 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:16:50,831 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:16:51,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:16:56,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:01,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:01,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:01,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:02,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:07,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:11,638 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:17:11,640 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:17:11,640 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:17:11,640 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:17:11,839 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:17:13,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:14,840 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:17:15,097 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:17:16,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:16,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:16,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:18,544 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:23,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:28,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:31,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:31,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:31,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:34,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:39,259 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:17:39,261 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:17:39,261 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:17:39,262 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:17:39,851 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:17:39,996 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:42,852 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:17:44,997 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:45,098 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:17:46,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:17:46,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:17:46,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:17:50,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:17:55,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:00,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:01,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:01,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:01,540 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:06,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:06,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:18:06,621 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:18:06,621 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:18:06,622 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:18:06,861 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:18:08,862 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:18:12,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:15,099 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:18:16,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:16,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:16,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:17,552 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:22,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:27,554 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:31,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:31,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:31,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:31,955 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:18:31,956 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:18:31,956 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:18:31,957 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:18:32,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:32,872 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:18:34,873 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:18:37,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:42,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:45,100 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:18:46,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:18:46,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:18:46,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:18:48,589 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:53,590 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:18:53,851 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:18:53,852 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:18:53,853 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:18:53,853 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:18:53,881 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:18:56,883 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:18:59,583 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:01,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:01,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:01,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:05,552 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:10,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:15,101 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:19:16,102 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:16,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:16,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:16,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:18,398 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:19:18,399 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:19:18,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:19:18,400 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:19:18,893 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:19:20,894 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:19:21,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:26,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:31,268 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:31,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:31,500 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:31,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:36,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:41,006 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:19:41,008 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:19:41,008 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:19:41,009 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:19:41,791 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:41,902 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:19:42,902 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:19:45,102 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:19:46,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:19:46,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:19:46,504 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:19:47,605 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:52,606 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:19:57,608 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:01,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:01,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:01,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:02,608 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:07,563 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:20:07,564 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:20:07,564 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:20:07,564 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:20:07,911 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:20:08,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:10,913 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:20:13,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:15,104 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:20:16,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:16,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:16,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:18,536 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:23,537 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:28,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:31,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:31,501 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:31,536 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:31,759 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:20:31,760 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:20:31,760 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:20:31,761 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:20:31,920 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:20:34,479 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:34,921 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:20:39,480 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:44,481 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:45,105 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:20:46,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:20:46,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:20:46,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:20:49,567 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:54,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:20:54,561 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:20:54,561 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:20:54,562 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:20:54,929 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:20:55,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:20:56,929 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:21:00,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:01,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:01,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:01,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:05,621 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:10,622 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:15,106 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:21:16,107 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:16,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:16,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:16,542 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:17,495 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:21:17,497 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:21:17,497 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:21:17,497 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:21:17,938 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:21:20,939 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:21:21,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:26,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:31,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:31,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:31,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:31,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:36,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:41,552 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:45,107 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:21:45,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:21:45,159 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:21:45,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:21:45,160 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:21:45,950 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:21:46,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:21:46,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:21:46,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:21:46,950 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:21:47,536 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:52,537 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:21:57,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:01,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:01,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:01,542 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:02,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:07,423 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:22:07,425 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:22:07,425 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:22:07,426 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:22:07,958 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:22:08,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:08,959 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:22:13,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:15,109 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:22:16,502 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:16,503 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:16,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:18,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:23,579 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:28,580 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:31,337 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:22:31,338 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:22:31,338 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:22:31,339 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:22:31,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:31,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:31,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:31,968 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:22:32,969 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:22:34,066 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:39,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:44,069 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:45,109 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:22:46,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:22:46,512 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:22:46,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:22:49,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:54,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:22:56,698 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:22:56,699 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:22:56,700 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:22:56,700 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:22:56,978 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:22:58,979 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:23:00,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:01,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:01,512 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:01,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:05,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:10,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:15,111 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:23:16,112 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:16,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:16,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:16,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:21,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:22,901 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:23:22,902 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:23:22,902 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:23:22,903 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:23:22,989 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:23:24,989 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:23:26,647 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:31,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:31,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:31,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:32,639 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:37,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:42,641 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:45,112 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:23:46,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:23:46,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:23:46,553 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:23:46,704 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:23:46,705 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:23:46,705 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:23:46,706 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:23:47,000 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:23:48,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:49,000 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:23:53,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:23:58,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:01,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:01,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:01,513 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:03,586 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:08,587 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:09,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:24:09,546 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:24:09,547 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:24:09,547 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:24:10,010 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:24:13,011 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:24:14,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:15,113 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:24:16,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:16,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:16,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:19,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:24,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:29,546 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:31,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:31,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:31,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:34,465 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:24:34,466 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:24:34,466 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:24:34,467 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:24:35,021 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:24:35,195 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:37,022 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:24:40,196 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:45,114 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:24:46,115 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:46,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:24:46,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:24:46,518 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:24:51,649 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:24:56,650 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:01,392 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:25:01,393 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:25:01,393 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:25:01,394 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:25:01,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:01,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:01,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:02,032 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:25:02,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:03,032 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:25:07,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:12,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:15,116 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:25:16,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:16,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:16,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:17,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:22,561 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:26,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:25:26,745 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:25:26,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:25:26,746 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:25:27,043 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:25:28,462 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:29,044 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:25:31,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:31,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:31,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:33,635 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:38,636 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:43,637 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:45,117 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:25:46,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:25:46,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:25:46,554 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:25:47,103 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:25:47,105 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:25:47,105 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:25:47,106 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:25:48,052 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:25:48,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:49,053 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:25:53,866 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:25:58,867 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:01,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:01,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:01,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:04,644 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:09,645 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:12,782 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:26:12,783 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:26:12,783 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:26:12,784 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:26:13,062 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:26:15,062 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:26:15,117 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:26:15,119 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:16,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:16,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:16,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:20,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:25,619 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:30,621 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:31,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:31,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:31,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:34,255 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:26:34,257 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:26:34,257 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:26:34,257 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:26:35,071 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:26:36,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:37,072 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:26:41,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:45,119 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:26:46,119 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:46,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:26:46,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:26:46,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:26:51,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:56,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:26:58,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:26:58,965 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:26:58,965 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:26:58,965 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:26:59,081 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:27:01,082 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:27:01,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:01,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:01,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:01,633 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:06,634 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:11,635 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:15,119 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:27:16,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:16,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:16,556 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:17,597 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:22,598 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:26,555 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:27:26,557 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:27:26,557 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:27:26,558 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:27:27,093 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:27:28,303 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:29,094 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:27:31,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:31,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:31,518 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:33,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:38,619 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:43,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:45,121 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:27:46,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:27:46,515 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:27:46,556 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:27:49,565 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:27:51,496 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:27:51,497 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:27:51,498 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:27:51,498 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:27:52,103 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:27:53,104 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:27:55,234 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:00,235 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:01,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:01,516 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:01,519 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:05,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:10,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:15,082 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:28:15,083 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:28:15,084 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:28:15,084 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:28:15,113 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:28:15,122 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:28:15,787 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:16,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:16,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:16,517 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:17,114 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:28:21,642 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:26,643 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:31,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:31,526 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:31,566 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:32,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:37,572 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:41,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:28:41,269 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:28:41,269 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:28:41,269 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:28:42,124 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:28:42,998 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:43,124 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:28:45,123 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:28:46,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:28:46,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:28:46,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:28:48,566 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:53,567 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:28:58,568 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:01,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:01,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:01,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:03,589 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:05,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:29:05,160 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:29:05,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:29:05,161 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:29:06,134 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:29:07,135 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:29:08,989 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:13,991 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:15,124 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:29:16,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:16,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:16,522 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:19,600 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:24,601 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:29,602 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:31,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:31,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:31,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:32,263 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:29:32,264 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:29:32,264 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:29:32,265 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:29:33,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:29:35,049 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:35,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:29:40,050 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:45,051 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:45,125 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:29:46,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:29:46,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:29:46,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:29:50,564 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:55,565 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:29:55,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:29:55,762 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:29:55,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:29:55,763 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:29:56,154 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:29:57,154 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:30:01,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:01,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:01,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:01,522 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:06,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:11,579 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:15,127 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:30:16,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:16,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:16,521 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:16,640 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:21,641 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:25,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:30:25,140 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:30:25,140 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:30:25,141 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:30:25,166 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:30:26,878 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:27,167 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:30:31,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:31,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:31,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:32,557 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:37,559 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:42,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:45,128 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:30:46,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:30:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:30:46,522 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:30:47,628 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:48,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:30:48,430 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:30:48,430 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:30:48,431 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:30:49,176 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:30:51,177 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:30:53,277 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:30:58,278 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:01,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:03,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:08,572 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:11,181 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:31:11,183 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:31:11,183 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:31:11,183 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:31:11,185 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:31:13,185 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:31:13,902 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:15,129 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:31:16,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:16,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:19,605 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:24,606 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:29,607 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:31,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:31,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:31,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:34,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:31:34,408 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:31:34,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:31:34,409 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:31:35,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:35,194 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:31:37,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:31:40,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:45,123 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:45,130 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:31:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:31:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:31:46,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:31:50,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:55,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:31:58,282 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:31:58,284 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:31:58,284 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:31:58,285 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:31:59,203 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:32:01,003 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:01,204 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:32:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:01,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:01,564 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:06,646 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:11,647 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:15,132 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:32:16,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:16,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:17,614 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:22,615 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:22,957 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:32:22,959 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:32:22,959 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:32:22,960 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:32:23,213 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:32:25,214 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:32:27,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:31,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:31,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:31,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:33,614 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:38,615 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:43,616 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:45,133 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:32:46,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:32:46,437 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:32:46,437 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:32:46,437 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:32:46,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:32:46,523 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:32:46,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:32:47,224 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:32:49,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:49,225 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:32:54,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:32:59,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:01,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:01,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:04,594 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:09,595 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:09,888 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:33:09,894 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:33:09,894 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:33:09,894 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:33:10,234 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:33:11,234 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:33:14,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:15,134 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:33:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:16,523 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:16,524 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:20,569 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:25,570 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:30,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:31,336 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:33:31,338 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:33:31,338 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:33:31,338 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:33:31,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:31,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:31,798 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:32,242 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:33:33,243 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:33:36,079 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:41,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:45,135 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:33:46,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:46,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:33:46,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:33:46,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:33:51,831 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:56,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:33:58,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:33:58,389 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:33:58,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:33:58,390 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:33:59,253 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:34:01,254 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:34:01,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:01,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:01,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:02,806 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:07,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:12,808 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:15,136 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:34:16,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:16,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:16,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:17,897 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:22,665 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:34:22,666 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:34:22,666 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:34:22,666 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:34:23,263 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:34:23,439 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:25,264 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:34:28,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:31,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:31,761 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:31,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:33,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:38,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:43,827 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:45,137 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:34:46,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:34:46,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:34:46,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:34:48,192 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:34:48,193 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:34:48,193 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:34:48,194 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:34:48,272 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:34:48,995 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:49,273 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:34:53,996 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:34:58,997 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:01,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:01,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:01,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:04,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:09,826 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:10,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:35:10,349 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:35:10,350 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:35:10,350 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:35:11,283 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:35:11,283 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:35:15,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:15,138 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:35:16,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:16,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:16,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:20,874 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:25,875 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:30,876 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:31,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:31,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:31,790 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:34,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:35:34,355 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:35:34,356 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:35:34,356 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:35:35,294 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:35:35,294 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:35:36,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:41,102 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:45,139 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:35:46,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:46,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:35:46,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:35:46,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:35:51,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:35:56,843 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:01,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:36:01,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:36:01,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:36:02,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:02,977 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:36:02,978 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:36:02,978 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:36:02,979 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:36:03,305 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:36:05,306 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:36:08,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:13,154 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:15,140 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:36:16,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:36:16,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:36:16,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:36:18,829 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:23,830 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:25,212 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:36:25,213 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:36:25,213 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:36:25,214 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:36:25,315 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:36:28,316 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:36:28,981 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:31,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:36:31,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:36:31,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:36:34,845 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:39,846 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:44,847 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:45,141 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:36:46,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:36:46,762 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:36:46,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:36:49,870 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:50,279 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:36:50,281 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:36:50,281 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:36:50,281 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:36:50,324 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:36:52,325 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:36:54,992 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:36:59,993 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:01,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:37:01,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:37:01,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:37:05,875 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:10,876 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:13,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:37:13,749 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:37:13,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:37:13,750 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:37:14,333 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:37:15,142 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:37:16,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:16,334 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:37:16,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:37:16,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:37:16,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:37:21,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:26,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:31,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:37:31,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:37:31,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:37:32,826 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:35,120 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:37:35,121 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:37:35,121 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:37:35,122 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:37:35,342 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:37:37,887 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:38,344 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:37:42,888 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:45,144 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:37:46,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:37:46,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:37:46,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:37:48,881 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:53,882 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:58,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:37:59,764 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:37:59,765 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:37:59,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:37:59,766 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:38:00,352 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:38:01,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:38:01,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:38:01,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:38:02,353 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:38:04,837 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:09,838 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:14,839 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:15,145 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:38:16,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:38:16,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:38:16,799 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:38:20,800 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:24,863 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:38:24,864 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:38:24,864 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:38:24,864 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:38:25,363 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:38:26,607 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:28,364 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:38:31,608 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:31,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:38:31,764 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:38:31,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:38:36,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:41,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:45,146 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:38:46,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:38:46,764 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:38:46,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:38:46,893 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:48,975 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:38:48,976 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:38:48,977 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:38:48,977 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:38:49,372 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:38:52,373 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:38:52,672 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:38:57,673 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:01,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:39:01,764 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:39:01,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:39:02,875 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:07,876 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:12,877 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:13,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:39:13,170 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:39:13,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:39:13,171 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:39:13,382 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:39:15,147 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:39:16,384 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:39:16,764 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:39:16,764 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:39:16,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:39:17,889 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:22,890 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:27,891 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:31,764 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:39:31,764 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:39:31,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:39:33,873 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:36,574 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:39:36,575 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:39:36,575 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:39:36,576 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:39:37,391 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:39:38,392 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:39:39,284 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:44,286 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:45,149 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:39:46,764 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:39:46,764 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:39:46,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:39:49,882 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:54,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:39:59,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:01,154 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:40:01,155 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:40:01,155 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:40:01,156 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:40:01,402 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:40:01,777 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:40:01,778 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:40:01,781 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:40:04,403 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:40:05,060 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:10,061 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:15,061 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:15,150 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:40:16,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:40:16,765 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:40:16,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:40:20,854 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:25,855 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:26,264 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:40:26,265 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:40:26,266 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:40:26,266 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:40:26,412 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:40:28,413 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:40:30,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:31,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:40:31,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:40:31,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:40:36,894 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:41,895 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:45,151 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:40:46,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:40:46,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:40:46,806 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:40:46,899 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:48,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:40:48,312 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:40:48,312 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:40:48,313 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:40:48,420 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:40:50,421 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:40:52,198 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:40:57,199 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:01,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:41:01,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:41:01,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:41:02,945 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:07,945 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:12,947 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:13,252 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:41:13,253 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:41:13,253 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:41:13,254 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:41:13,430 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:41:15,152 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:41:16,431 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:41:16,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:41:16,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:41:16,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:41:18,886 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:23,887 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:28,887 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:31,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:41:31,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:41:31,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:41:34,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:36,995 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:41:36,996 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:41:36,996 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:41:36,997 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:41:37,440 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:41:40,441 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:41:40,709 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:45,153 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:41:46,154 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:46,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:41:46,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:41:46,781 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:41:51,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:41:56,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:01,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:42:01,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:42:01,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:42:01,853 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:02,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:42:02,729 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:42:02,729 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:42:02,730 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:42:03,451 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:42:06,452 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:42:07,451 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:12,452 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:15,154 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:42:16,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:42:16,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:42:16,781 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:42:17,872 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:22,873 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:26,718 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:42:26,719 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:42:26,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:42:26,720 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:42:27,460 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:42:28,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:30,461 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:42:31,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:42:31,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:42:31,781 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:42:33,809 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:38,810 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:43,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:45,155 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:42:46,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:42:46,767 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:42:46,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:42:48,890 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:53,891 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:42:56,241 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:42:56,242 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:42:56,242 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:42:56,243 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:42:56,472 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:42:58,473 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:42:58,962 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:01,766 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:43:01,767 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:43:01,781 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:43:04,837 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:09,838 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:14,839 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:15,156 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:43:16,767 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:43:16,767 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:43:16,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:43:19,887 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:23,585 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:43:23,586 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:43:23,586 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:43:23,587 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:43:24,484 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:43:25,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:26,485 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:43:30,341 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:31,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:43:31,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:43:31,782 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:43:35,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:40,843 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:45,157 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:43:46,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:46,767 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:43:46,767 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:43:46,808 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:43:48,739 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:43:48,743 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:43:48,743 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:43:48,744 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:43:49,494 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:43:51,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:43:52,495 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:43:56,510 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:01,511 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:01,767 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:44:01,767 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:44:01,782 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:44:06,849 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:11,850 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:14,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:44:14,137 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:44:14,138 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:44:14,138 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:44:14,505 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:44:15,158 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:44:16,506 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:44:16,767 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:44:16,768 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:44:16,782 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:44:17,847 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:22,848 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:27,849 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:31,767 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:44:31,768 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:44:31,808 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:44:33,813 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:38,814 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:40,328 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:44:40,329 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:44:40,329 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:44:40,330 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:44:40,515 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:44:42,516 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:44:44,059 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:45,159 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:44:46,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:44:46,768 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:44:46,782 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:44:49,858 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:54,859 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:44:59,860 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:01,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:45:01,768 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:45:01,809 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:45:02,202 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:45:02,203 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:45:02,203 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:45:02,204 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:45:02,526 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:45:04,527 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:45:05,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:10,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:15,022 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:15,160 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:45:16,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:45:16,768 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:45:16,783 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:45:20,881 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:25,882 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:27,059 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:45:27,060 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:45:27,060 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:45:27,061 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:45:27,538 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:45:30,540 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:45:30,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:31,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:45:31,768 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:45:31,783 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:45:35,898 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:40,899 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:45,161 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:45:46,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:46,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:45:46,769 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:45:46,809 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:45:50,526 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:45:50,528 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:45:50,528 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:45:50,528 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:45:50,547 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:45:51,529 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:45:54,549 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:45:56,626 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:01,627 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:01,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:46:01,769 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:46:01,783 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:46:06,848 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:11,849 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:15,162 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:46:16,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:46:16,769 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:46:16,809 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:46:16,871 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:17,084 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:46:17,086 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:46:17,086 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:46:17,087 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:46:17,559 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:46:20,560 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:46:22,821 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:27,822 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:31,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:46:31,769 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:46:31,783 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:46:33,800 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:38,800 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:43,075 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:46:43,076 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:46:43,076 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:46:43,077 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:46:43,570 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:46:43,812 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:45,163 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:46:46,571 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:46:46,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:46:46,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:46:46,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:46:48,858 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:53,859 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:46:58,860 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:01,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:47:01,769 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:47:01,810 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:47:04,855 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:06,462 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:47:06,463 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:47:06,464 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:47:06,464 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:47:06,579 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:47:08,580 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:47:10,190 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:15,165 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:47:16,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:16,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:47:16,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:47:16,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:47:21,821 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:26,822 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:30,373 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:47:30,374 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:47:30,374 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:47:30,375 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:47:30,588 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:47:31,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:47:31,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:47:31,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:47:31,881 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:32,589 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:47:36,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:41,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:45,165 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:47:46,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:47:46,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:47:46,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:47:47,794 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:52,795 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:56,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:47:56,434 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:47:56,435 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:47:56,435 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:47:56,600 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:47:58,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:47:58,601 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:48:01,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:48:01,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:48:01,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:48:03,830 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:08,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:13,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:15,166 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:48:16,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:48:16,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:48:16,810 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:48:18,851 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:20,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:48:20,055 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:48:20,056 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:48:20,056 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:48:20,609 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:48:22,610 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:48:24,755 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:29,756 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:31,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:48:31,770 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:48:31,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:48:34,880 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:39,881 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:44,882 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:45,167 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:48:45,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:48:45,866 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:48:45,867 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:48:45,867 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:48:46,619 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:48:46,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:48:46,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:48:46,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:48:48,620 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:48:49,889 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:54,890 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:48:59,891 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:01,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:49:01,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:49:01,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:49:05,874 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:10,875 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:12,018 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:49:12,019 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:49:12,019 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:49:12,020 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:49:12,630 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:49:14,631 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:49:15,168 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:49:16,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:16,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:49:16,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:49:16,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:49:21,818 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:26,819 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:31,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:49:31,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:49:31,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:49:31,894 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:36,895 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:40,873 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:49:40,875 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:49:40,875 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:49:40,876 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:49:41,641 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:49:42,597 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:42,641 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:49:45,169 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:49:46,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:49:46,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:49:46,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:49:47,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:52,804 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:49:57,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:01,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:50:01,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:50:01,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:50:02,851 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:03,031 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:50:03,032 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:50:03,032 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:50:03,033 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:50:03,649 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:50:04,649 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:50:08,750 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:13,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:15,170 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:50:16,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:50:16,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:50:16,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:50:18,890 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:23,892 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:25,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:50:25,884 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:50:25,884 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:50:25,885 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:50:26,658 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:50:28,659 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:50:29,584 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:31,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:50:31,771 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:50:31,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:50:34,900 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:39,901 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:44,902 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:45,171 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:50:46,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:50:46,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:50:46,812 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:50:48,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:50:48,390 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:50:48,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:50:48,390 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:50:48,666 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:50:50,124 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:50:50,666 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:50:55,125 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:00,126 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:01,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:51:01,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:51:01,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:51:05,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:10,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:13,568 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:51:13,569 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:51:13,570 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:51:13,570 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:51:13,679 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:51:15,173 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:51:16,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:16,682 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:51:16,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:51:16,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:51:16,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:51:21,858 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:26,859 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:31,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:51:31,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:51:31,812 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:51:31,885 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:36,175 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:51:36,176 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:51:36,177 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:51:36,177 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:51:36,696 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:51:37,178 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:38,697 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:51:42,833 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:45,173 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:51:46,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:51:46,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:51:46,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:51:47,886 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:52,887 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:57,888 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:51:58,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:51:58,633 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:51:58,633 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:51:58,634 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:51:58,705 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:52:00,705 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:52:01,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:52:01,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:52:01,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:52:02,895 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:07,896 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:12,897 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:15,174 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:52:16,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:52:16,772 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:52:16,813 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:52:18,864 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:22,006 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:52:22,007 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:52:22,007 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:52:22,008 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:52:22,715 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:52:24,033 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:24,716 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:52:29,034 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:31,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:52:31,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:52:31,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:52:34,901 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:39,902 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:44,903 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:45,175 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:52:46,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:52:46,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:52:46,813 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:52:47,333 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:52:47,334 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:52:47,334 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:52:47,335 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:52:47,729 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:52:50,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:52:50,730 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:52:55,090 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:00,091 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:01,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:53:01,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:53:01,787 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:53:05,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:10,806 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:10,829 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:53:10,830 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:53:10,830 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:53:10,831 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:53:11,739 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:53:12,740 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:53:15,176 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:53:16,177 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:16,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:53:16,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:53:16,787 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:53:21,874 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:26,875 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:31,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:53:31,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:53:31,813 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:53:31,890 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:34,930 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:53:34,932 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:53:34,932 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:53:34,932 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:53:35,755 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:53:36,755 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:53:37,647 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:42,648 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:45,177 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:53:46,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:53:46,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:53:46,794 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:53:47,864 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:52,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:57,866 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:53:58,240 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:53:58,241 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:53:58,242 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:53:58,242 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:53:58,769 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:54:00,772 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:54:01,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:54:01,773 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:54:01,792 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:54:03,835 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:08,836 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:13,837 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:15,178 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:54:16,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:54:16,774 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:54:16,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:54:19,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:21,416 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:54:21,418 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:54:21,418 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:54:21,418 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:54:21,781 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:54:24,782 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:54:25,449 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:30,450 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:31,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:54:31,774 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:54:31,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:54:35,914 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:40,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:44,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:54:44,315 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:54:44,315 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:54:44,316 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:54:44,791 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:54:45,179 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:54:46,180 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:46,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:54:46,774 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:54:46,792 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:54:46,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:54:51,863 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:54:56,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:01,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:55:01,774 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:55:01,800 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:55:02,801 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:07,581 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:55:07,582 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:55:07,582 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:55:07,583 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:55:07,800 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:55:08,297 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:10,801 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:55:13,298 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:15,180 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:55:16,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:55:16,774 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:55:16,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:55:18,810 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:23,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:28,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:31,215 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:55:31,216 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:55:31,216 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:55:31,217 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:55:31,809 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:55:31,920 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:55:31,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:55:31,922 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:55:32,809 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:55:33,957 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:38,959 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:43,959 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:45,182 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:55:46,915 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:55:46,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:55:46,916 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:55:49,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:55:52,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:55:52,986 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:55:52,987 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:55:52,987 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:55:53,816 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:55:54,817 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:55:55,707 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:00,708 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:01,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:56:01,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:56:01,917 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:56:05,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:10,986 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:15,183 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:56:16,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:16,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:56:16,744 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:56:16,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:56:16,745 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:56:16,825 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:56:17,198 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:56:17,199 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:56:17,199 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:56:18,826 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:56:21,467 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:26,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:31,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:31,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:56:32,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:56:32,158 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:56:37,303 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:38,997 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:56:38,999 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:56:38,999 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:56:38,999 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:56:39,837 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:56:40,838 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:56:42,697 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:45,184 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:56:46,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:56:47,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:56:47,158 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:56:48,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:53,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:56:58,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:01,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:57:02,101 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:57:02,102 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:57:02,103 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:57:02,103 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:57:02,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:57:02,157 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:57:02,845 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:57:03,859 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:04,847 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:57:08,860 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:13,861 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:15,185 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:57:16,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:57:17,157 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:57:17,158 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:57:19,265 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:24,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:29,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:30,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:57:30,443 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:57:30,443 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:57:30,444 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:57:30,859 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:57:31,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:57:32,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:57:32,159 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:57:32,859 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:57:35,249 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:40,250 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:45,186 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:57:46,187 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:46,916 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:57:47,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:57:47,158 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:57:51,237 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:53,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:57:53,788 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:57:53,788 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:57:53,788 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:57:53,867 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:57:56,544 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:57:56,868 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:58:01,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:01,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:58:02,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:58:02,158 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:58:07,279 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:12,280 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:15,187 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:58:15,458 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:58:15,459 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:58:15,459 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:58:15,460 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:58:15,876 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:58:16,877 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:58:16,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:58:17,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:58:17,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:58:18,207 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:23,208 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:28,209 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:31,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:58:32,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:58:32,159 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:58:34,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:37,707 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:58:37,708 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:58:37,708 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:58:37,709 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:58:37,886 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:58:39,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:40,887 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:58:44,579 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:45,188 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:58:46,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:58:47,158 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:58:47,159 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:58:50,213 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:58:55,214 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:00,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:59:00,055 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:59:00,055 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:59:00,055 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:59:00,789 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:00,894 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:59:01,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:59:02,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:59:02,159 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:59:02,895 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:59:06,192 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:11,193 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:15,189 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:59:16,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:59:17,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:17,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:59:17,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:59:22,191 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:26,342 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:59:26,343 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:59:26,343 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:59:26,344 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:59:26,904 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:59:28,068 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:28,904 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:59:31,917 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:59:32,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:59:32,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:59:33,201 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:38,202 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:43,203 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:45,190 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 08:59:46,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 08:59:47,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 08:59:47,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 08:59:48,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:53,297 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 08:59:54,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 08:59:54,126 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 08:59:54,127 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 08:59:54,127 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 08:59:54,915 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 08:59:56,918 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 08:59:58,863 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:01,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:00:02,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:00:02,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:00:04,240 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:09,241 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:14,242 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:15,190 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:00:16,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:00:17,125 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:00:17,126 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:00:17,127 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:00:17,127 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:00:17,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:00:17,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:00:17,927 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:00:19,852 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:19,927 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:00:24,853 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:29,854 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:31,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:00:32,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:00:32,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:00:35,241 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:40,242 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:41,055 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:00:41,057 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:00:41,057 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:00:41,058 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:00:41,935 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:00:43,936 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:00:45,192 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:00:46,193 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:46,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:00:47,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:00:47,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:00:51,265 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:00:56,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:01,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:01,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:01:02,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:01:02,160 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:01:07,203 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:08,940 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:01:08,941 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:01:08,942 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:01:08,942 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:01:08,945 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:01:11,946 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:01:12,739 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:15,193 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:01:16,918 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:01:17,161 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:01:17,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:01:18,221 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:23,222 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:28,223 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:31,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:01:32,161 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:01:32,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:01:33,273 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:33,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:01:33,297 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:01:33,297 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:01:33,298 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:01:33,954 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:01:35,955 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:01:38,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:43,341 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:45,194 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:01:46,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:01:47,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:01:47,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:01:49,277 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:54,278 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:01:59,279 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:00,041 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:02:00,042 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:02:00,042 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:02:00,043 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:02:00,966 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:02:01,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:02:02,160 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:02:02,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:02:03,968 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:02:05,258 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:10,259 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:15,195 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:02:16,196 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:16,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:02:17,161 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:02:17,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:02:21,277 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:21,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:02:21,398 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:02:21,398 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:02:21,399 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:02:21,976 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:02:23,977 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:02:27,109 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:31,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:02:32,110 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:32,161 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:02:32,161 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:02:37,271 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:42,272 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:43,942 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:02:43,943 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:02:43,944 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:02:43,944 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:02:43,984 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:02:45,196 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:02:46,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:02:47,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:02:47,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:02:47,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:47,986 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:02:52,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:02:57,294 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:01,919 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:03:02,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:03:02,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:03:03,211 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:07,983 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:03:07,984 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:03:07,984 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:03:07,985 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:03:07,994 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:03:08,705 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:09,995 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:03:13,706 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:15,197 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:03:16,920 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:03:17,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:03:17,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:03:19,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:24,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:29,268 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:31,920 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:03:32,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:03:32,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:03:33,018 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:03:33,019 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:03:33,019 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:03:33,020 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:03:34,005 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:03:34,905 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:36,006 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:03:39,906 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:44,907 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:45,197 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:03:46,920 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:03:47,161 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:03:47,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:03:50,248 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:03:55,249 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:00,123 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:04:00,124 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:04:00,124 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:04:00,125 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:04:00,864 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:01,015 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:04:01,920 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:04:02,015 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:04:02,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:04:02,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:04:06,265 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:11,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:15,198 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:04:16,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:04:17,161 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:17,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:04:17,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:04:22,222 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:26,634 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:04:26,635 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:04:26,636 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:04:26,636 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:04:27,025 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:04:27,352 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:30,027 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:04:31,920 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:04:32,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:04:32,163 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:04:33,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:38,294 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:43,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:45,200 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:04:46,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:04:47,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:04:47,162 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:04:48,271 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:04:48,272 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:04:48,273 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:04:48,273 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:04:49,027 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:49,033 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:04:52,035 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:04:54,028 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:04:59,029 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:01,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:05:02,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:05:02,163 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:05:04,570 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:09,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:14,053 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:05:14,054 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:05:14,055 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:05:14,055 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:05:14,830 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:15,044 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:05:15,201 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:05:16,044 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:05:16,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:05:17,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:05:17,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:05:20,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:25,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:30,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:31,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:05:32,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:05:32,165 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:05:35,397 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:36,590 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:05:36,591 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:05:36,591 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:05:36,591 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:05:37,053 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:05:40,055 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:05:41,319 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:45,201 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:05:46,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:05:47,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:47,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:05:47,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:05:52,298 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:05:57,299 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:01,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:06:02,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:06:02,165 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:06:02,403 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:06:02,404 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:06:02,404 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:06:02,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:02,405 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:06:03,064 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:06:06,067 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:06:08,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:13,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:15,203 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:06:16,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:06:17,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:06:17,165 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:06:18,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:23,276 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:26,213 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:06:26,214 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:06:26,214 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:06:26,215 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:06:27,078 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:06:28,078 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:06:28,921 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:31,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:06:32,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:06:32,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:06:34,303 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:39,304 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:44,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:45,204 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:06:46,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:06:47,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:06:47,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:06:49,351 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:06:49,352 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:06:49,352 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:06:49,353 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:06:49,353 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:06:50,089 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:06:52,090 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:06:55,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:00,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:01,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:07:02,165 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:07:02,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:07:05,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:10,294 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:12,932 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:07:12,934 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:07:12,934 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:07:12,935 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:07:13,097 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:07:15,204 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:07:16,099 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:07:16,205 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:16,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:07:17,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:07:17,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:07:21,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:26,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:31,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:31,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:07:32,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:07:32,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:07:34,686 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:07:34,687 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:07:34,687 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:07:34,688 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:07:35,106 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:07:36,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:38,108 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:07:41,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:45,206 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:07:46,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:07:47,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:47,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:07:47,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:07:52,249 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:57,251 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:07:59,508 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:07:59,509 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:07:59,509 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:07:59,510 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:08:00,115 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:08:01,922 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:08:02,116 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:08:02,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:08:02,167 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:08:02,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:07,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:12,294 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:15,207 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:08:16,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:08:17,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:08:17,167 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:08:18,284 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:23,285 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:25,669 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:08:25,670 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:08:25,670 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:08:25,670 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:08:26,124 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:08:28,125 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:08:28,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:31,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:08:32,166 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:08:32,167 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:08:34,225 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:39,226 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:44,227 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:45,212 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:08:46,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:08:47,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:08:47,167 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:08:49,248 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:49,857 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:08:49,858 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:08:49,858 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:08:49,859 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:08:50,135 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:08:52,135 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:08:54,584 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:08:59,585 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:01,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:09:02,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:09:02,167 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:09:05,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:10,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:14,503 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:09:14,504 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:09:14,504 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:09:14,504 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:09:15,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:09:15,209 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:09:15,225 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:16,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:09:16,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:09:17,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:09:17,167 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:09:20,288 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:25,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:30,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:31,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:09:32,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:09:32,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:09:36,232 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:38,946 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:09:38,948 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:09:38,948 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:09:38,948 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:09:39,154 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:09:42,114 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:42,155 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:09:45,210 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:09:46,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:09:47,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:47,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:09:47,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:09:52,277 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:09:57,278 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:01,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:10:02,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:10:02,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:10:03,274 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:04,947 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:10:04,948 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:10:04,948 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:10:04,950 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:10:05,165 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:10:08,166 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:10:08,686 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:13,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:15,211 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:10:16,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:10:17,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:10:17,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:10:19,216 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:24,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:29,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:29,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:10:29,469 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:10:29,469 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:10:29,469 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:10:30,176 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:10:31,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:10:32,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:10:32,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:10:32,177 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:10:34,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:39,306 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:44,307 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:45,212 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:10:46,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:10:47,167 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:10:47,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:10:50,253 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:53,741 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:10:53,742 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:10:53,742 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:10:53,743 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:10:54,185 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:10:55,488 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:10:56,186 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:11:00,489 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:01,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:11:02,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:11:02,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:11:06,264 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:11,265 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:15,213 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:11:16,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:11:17,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:17,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:11:17,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:11:19,713 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:11:19,714 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:11:19,714 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:11:19,715 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:11:20,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:11:22,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:11:22,673 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:27,674 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:31,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:11:32,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:11:32,168 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:11:33,223 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:38,224 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:43,225 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:44,663 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:11:44,664 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:11:44,664 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:11:44,665 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:11:45,205 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:11:45,214 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:11:46,206 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:11:46,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:11:47,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:11:47,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:11:49,193 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:54,194 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:11:59,195 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:01,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:12:02,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:12:02,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:12:05,022 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:10,023 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:11,144 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:12:11,145 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:12:11,146 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:12:11,146 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:12:11,216 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:12:14,218 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:12:15,215 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:12:15,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:16,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:12:17,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:12:17,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:12:20,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:25,297 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:30,298 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:31,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:12:32,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:12:32,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:12:32,367 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:12:32,368 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:12:32,368 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:12:32,369 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:12:33,226 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:12:34,227 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:12:36,096 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:41,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:45,216 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:12:46,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:46,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:12:47,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:12:47,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:12:51,223 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:56,224 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:12:57,451 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:12:57,452 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:12:57,453 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:12:57,453 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:12:58,236 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:13:00,236 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:13:01,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:13:02,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:02,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:13:02,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:13:07,215 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:12,216 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:15,218 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:13:16,926 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:13:17,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:13:17,169 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:13:17,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:22,297 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:23,645 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:13:23,646 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:13:23,647 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:13:23,647 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:13:24,246 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:13:26,247 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:13:27,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:31,932 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:13:32,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:13:32,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:13:33,253 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:38,251 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:43,252 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:45,219 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:13:46,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:13:47,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:13:47,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:13:48,298 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:48,482 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:13:48,484 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:13:48,484 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:13:48,484 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:13:49,262 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:13:50,262 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:13:54,243 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:13:59,244 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:01,930 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:14:02,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:14:02,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:14:05,234 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:10,235 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:14,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:14:14,316 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:14:14,316 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:14:14,317 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:14:15,220 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:14:15,280 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:14:16,221 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:16,281 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:14:16,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:14:17,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:14:17,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:14:21,281 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:26,281 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:31,282 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:31,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:14:32,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:14:32,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:14:36,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:40,877 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:14:40,878 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:14:40,878 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:14:40,878 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:14:41,292 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:14:41,580 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:42,292 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:14:45,221 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:14:46,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:14:47,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:47,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:14:47,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:14:52,290 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:14:57,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:01,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:15:02,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:15:02,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:15:03,264 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:05,029 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:15:05,031 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:15:05,031 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:15:05,031 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:15:05,301 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:15:08,302 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:15:08,783 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:13,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:15,222 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:15:16,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:15:17,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:15:17,170 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:15:19,236 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:24,237 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:29,055 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:15:29,056 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:15:29,056 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:15:29,057 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:15:29,309 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:15:29,749 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:30,310 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:15:31,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:15:32,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:15:32,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:15:35,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:40,221 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:45,222 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:45,223 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:15:46,932 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:15:47,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:15:47,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:15:50,302 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:53,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:15:53,786 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:15:53,786 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:15:53,786 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:15:54,318 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:15:55,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:15:56,319 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:16:00,526 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:01,932 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:16:02,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:16:02,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:16:06,236 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:11,237 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:15,224 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:16:16,244 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:16:16,245 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:16:16,246 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:16:16,246 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:16,247 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:16:16,326 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:16:16,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:16:17,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:16:17,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:16:18,327 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:16:21,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:26,297 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:31,298 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:31,948 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:16:32,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:16:32,172 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:16:37,258 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:40,757 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:16:40,758 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:16:40,758 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:16:40,759 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:16:41,336 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:16:42,336 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:16:42,481 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:45,225 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:16:46,948 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:16:47,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:16:47,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:16:48,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:53,296 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:16:58,297 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:01,948 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:17:02,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:17:02,171 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:17:04,230 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:09,231 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:09,378 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:17:09,379 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:17:09,379 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:17:09,379 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:17:10,348 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:17:12,348 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:17:15,124 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:15,226 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:17:16,949 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:17:17,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:17:17,172 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:17:20,216 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:25,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:30,218 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:31,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:17:32,171 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:17:32,172 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:17:34,333 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:17:34,334 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:17:34,334 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:17:34,335 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:17:34,357 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:17:36,046 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:36,357 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:17:41,047 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:45,232 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:17:46,233 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:46,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:17:47,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:17:47,172 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:17:52,227 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:17:57,228 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:00,679 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:18:00,680 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:18:00,681 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:18:00,681 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:18:01,366 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:18:01,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:18:02,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:18:02,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:18:02,240 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:02,367 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:18:07,241 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:12,242 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:15,228 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:18:16,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:18:17,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:18:17,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:18:17,286 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:22,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:26,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:18:26,744 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:18:26,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:18:26,746 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:18:27,376 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:18:27,482 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:28,376 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:18:31,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:18:32,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:18:32,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:18:33,269 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:38,270 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:43,271 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:45,230 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:18:46,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:18:47,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:18:47,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:18:48,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:51,236 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:18:51,237 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:18:51,238 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:18:51,238 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:18:51,384 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:18:52,385 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:18:54,011 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:18:59,012 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:01,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:19:02,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:19:02,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:19:04,244 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:09,244 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:14,246 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:15,231 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:19:15,497 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:19:15,498 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:19:15,499 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:19:15,499 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:19:16,393 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:19:16,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:19:17,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:19:17,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:19:18,394 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:19:19,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:24,294 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:29,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:31,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:19:32,172 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:19:32,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:19:34,303 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:38,753 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:19:38,754 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:19:38,754 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:19:38,755 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:19:39,400 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:19:39,543 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:40,400 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:19:44,544 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:45,231 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:19:46,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:19:47,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:19:47,173 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:19:50,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:19:55,276 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:00,277 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:01,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:20:02,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:20:02,174 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:20:02,518 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:20:02,519 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:20:02,520 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:20:02,520 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:20:03,410 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:20:04,410 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:20:06,231 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:11,232 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:15,232 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:20:16,233 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:16,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:20:17,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:20:17,174 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:20:22,228 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:27,230 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:29,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:20:29,054 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:20:29,054 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:20:29,054 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:20:29,420 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:20:30,420 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:20:31,951 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:20:32,173 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:20:32,174 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:20:32,273 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:37,274 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:42,274 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:45,233 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:20:46,952 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:20:47,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:20:47,174 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:20:48,202 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:53,203 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:20:53,472 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:20:53,474 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:20:53,474 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:20:53,474 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:20:54,430 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:20:56,430 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:20:58,221 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:01,952 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:21:02,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:21:02,174 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:21:04,219 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:09,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:14,220 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:15,234 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:21:15,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:21:15,534 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:21:15,534 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:21:15,534 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:21:16,437 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:21:16,952 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:21:17,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:21:17,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:21:18,438 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:21:19,231 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:24,232 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:29,232 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:31,952 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:21:32,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:21:32,174 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:21:34,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:37,735 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:21:37,737 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:21:37,737 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:21:37,737 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:21:38,445 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:21:39,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:40,446 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:21:44,470 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:45,235 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:21:46,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:21:47,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:21:47,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:21:50,215 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:21:55,216 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:00,217 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:01,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:22:02,072 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:22:02,074 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:22:02,074 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:22:02,074 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:22:02,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:22:02,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:22:02,456 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:22:04,457 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:22:05,570 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:10,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:15,237 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:22:16,238 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:16,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:22:17,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:22:17,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:22:22,208 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:26,555 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:22:26,557 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:22:26,557 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:22:26,557 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:22:27,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:27,465 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:22:28,465 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:22:31,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:22:32,174 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:22:32,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:22:33,246 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:38,247 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:43,248 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:45,238 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:22:46,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:22:47,175 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:22:47,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:22:48,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:52,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:22:52,151 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:22:52,151 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:22:52,151 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:22:52,474 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:22:53,885 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:22:55,475 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:22:58,886 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:01,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:23:02,175 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:23:02,175 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:23:04,269 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:09,270 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:14,271 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:15,239 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:23:16,868 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:23:16,869 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:23:16,870 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:23:16,870 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:23:16,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:23:17,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:23:17,330 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:23:17,484 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:23:19,485 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:23:19,603 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:24,604 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:29,605 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:31,953 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:23:32,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:23:32,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:23:35,329 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:37,613 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:23:37,614 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:23:37,614 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:23:37,614 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:23:38,492 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:23:39,492 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:23:40,365 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:45,240 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:23:46,241 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:46,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:23:47,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:23:47,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:23:51,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:23:56,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:01,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:01,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:24:02,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:24:02,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:24:03,796 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:24:03,797 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:24:03,798 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:24:03,798 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:24:04,504 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:24:06,532 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:07,505 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:24:11,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:15,242 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:24:16,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:24:17,243 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:17,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:24:17,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:24:22,322 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:27,323 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:28,604 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:24:28,605 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:24:28,605 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:24:28,606 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:24:29,513 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:24:31,514 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:24:31,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:24:32,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:24:32,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:24:32,422 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:37,422 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:42,424 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:45,243 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:24:46,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:24:47,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:24:47,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:24:48,403 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:53,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:24:54,651 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:24:54,652 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:24:54,652 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:24:54,653 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:24:55,523 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:24:57,524 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:24:59,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:01,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:25:02,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:25:02,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:25:04,403 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:09,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:14,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:15,244 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:25:16,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:25:17,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:25:17,294 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:25:19,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:19,873 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:25:19,874 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:25:19,875 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:25:19,875 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:25:20,534 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:25:23,535 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:25:24,615 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:29,616 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:31,954 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:25:32,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:25:32,293 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:25:35,349 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:40,349 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:45,245 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:25:45,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:25:45,632 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:25:45,632 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:25:45,632 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:45,633 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:25:46,543 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:25:46,956 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:25:47,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:25:47,293 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:25:47,544 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:25:51,384 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:25:56,385 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:01,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:01,956 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:26:02,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:26:02,293 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:26:06,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:09,413 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:26:09,414 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:26:09,414 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:26:09,414 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:26:09,554 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:26:11,555 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:26:12,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:15,246 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:26:16,956 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:26:17,247 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:17,293 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:26:17,293 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:26:22,347 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:27,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:31,952 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:26:31,953 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:26:31,953 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:26:31,954 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:26:31,976 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:26:32,430 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:32,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:26:32,431 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:26:32,563 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:26:35,564 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:26:37,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:42,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:45,251 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:26:46,962 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:26:47,385 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:26:47,386 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:26:48,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:53,458 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:56,447 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:26:56,449 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:26:56,449 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:26:56,449 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:26:56,572 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:26:59,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:26:59,573 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:27:01,962 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:27:02,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:27:02,386 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:27:04,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:09,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:14,437 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:15,248 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:27:16,962 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:27:17,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:27:17,386 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:27:19,447 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:23,601 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:27:23,602 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:27:23,602 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:27:23,604 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:27:24,583 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:27:25,366 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:25,583 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:27:30,367 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:31,962 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:27:32,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:27:32,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:27:35,452 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:40,454 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:45,249 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:27:46,250 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:46,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:27:47,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:27:47,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:27:48,585 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:27:48,586 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:27:48,586 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:27:48,587 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:27:48,593 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:27:51,314 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:27:51,595 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:27:56,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:01,316 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:01,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:28:02,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:28:02,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:28:06,467 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:11,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:11,950 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:28:11,951 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:28:11,952 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:28:11,954 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:28:12,604 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:28:13,604 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:28:15,250 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:28:16,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:28:17,251 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:17,386 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:28:17,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:28:22,486 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:27,487 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:31,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:28:32,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:28:32,388 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:28:32,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:36,108 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:28:36,109 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:28:36,109 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:28:36,109 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:28:36,614 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:28:37,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:39,615 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:28:42,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:45,252 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:28:46,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:28:47,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:28:47,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:28:48,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:53,430 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:58,430 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:28:58,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:28:58,470 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:28:58,470 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:28:58,471 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:28:58,622 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:29:01,623 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:29:01,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:29:02,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:29:02,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:29:03,467 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:08,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:13,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:15,252 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:29:16,963 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:29:17,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:29:17,387 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:29:19,427 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:22,334 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:29:22,336 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:29:22,336 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:29:22,336 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:29:22,631 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:29:25,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:25,632 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:29:30,052 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:31,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:29:32,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:29:32,388 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:29:35,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:40,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:45,253 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:29:46,254 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:46,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:29:47,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:29:47,388 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:29:49,853 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:29:49,854 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:29:49,855 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:29:49,855 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:29:50,642 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:29:51,577 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:29:51,643 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:29:56,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:01,578 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:01,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:30:02,387 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:30:02,388 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:30:07,450 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:12,450 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:13,571 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:30:13,572 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:30:13,572 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:30:13,573 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:30:13,651 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:30:15,254 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:30:15,652 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:30:16,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:30:17,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:30:17,388 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:30:17,456 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:22,456 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:27,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:31,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:30:32,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:30:32,388 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:30:33,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:37,941 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:30:37,942 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:30:37,942 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:30:37,943 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:30:38,662 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:30:38,684 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:39,663 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:30:43,685 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:45,255 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:30:46,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:30:47,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:30:47,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:30:49,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:54,526 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:30:59,527 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:01,234 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:31:01,235 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:31:01,235 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:31:01,236 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:31:01,672 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:31:02,022 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:31:02,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:31:02,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:31:03,673 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:31:05,425 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:10,426 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:15,256 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:31:16,257 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:17,019 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:31:17,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:31:17,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:31:21,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:26,508 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:30,248 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:31:30,249 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:31:30,249 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:31:30,250 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:31:30,683 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:31:31,683 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:31:31,994 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:32,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:31:32,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:31:32,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:31:37,442 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:42,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:45,257 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:31:47,019 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:31:47,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:31:47,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:31:47,485 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:52,486 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:56,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:31:56,723 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:31:56,724 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:31:56,724 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:31:57,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:31:57,693 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:31:59,694 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:32:02,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:32:02,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:32:02,389 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:32:03,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:08,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:13,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:15,258 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:32:17,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:32:17,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:32:17,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:32:18,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:23,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:32:23,277 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:32:23,277 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:32:23,277 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:32:23,703 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:32:24,043 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:25,704 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:32:29,044 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:32,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:32:32,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:32:32,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:32:34,453 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:39,454 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:44,455 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:44,660 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:32:44,661 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:32:44,662 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:32:44,662 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:32:44,711 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:32:45,260 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:32:47,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:32:47,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:32:47,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:32:47,712 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:32:49,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:54,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:32:59,470 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:02,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:33:02,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:33:02,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:33:05,426 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:07,176 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:33:07,177 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:33:07,177 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:33:07,178 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:33:07,720 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:33:09,721 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:33:11,191 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:15,260 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:33:16,261 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:17,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:33:17,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:33:17,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:33:21,526 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:26,527 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:30,615 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:33:30,616 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:33:30,617 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:33:30,617 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:33:30,729 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:33:32,020 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:33:32,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:33:32,391 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:33:32,423 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:33,730 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:33:37,425 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:42,426 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:45,262 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:33:47,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:33:47,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:33:47,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:33:47,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:52,511 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:56,795 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:33:56,796 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:33:56,797 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:33:56,797 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:33:57,657 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:33:57,741 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:33:59,742 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:34:02,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:34:02,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:34:02,390 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:34:03,489 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:08,490 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:13,491 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:15,263 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:34:17,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:34:17,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:34:17,391 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:34:18,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:19,658 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:34:19,659 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:34:19,659 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:34:19,660 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:34:19,751 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:34:21,752 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:34:24,416 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:29,417 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:32,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:34:32,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:34:32,391 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:34:34,470 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:39,471 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:42,155 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:34:42,157 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:34:42,157 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:34:42,158 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:34:42,760 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:34:43,760 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:34:44,897 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:45,264 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:34:47,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:34:47,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:34:47,391 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:34:50,508 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:34:55,509 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:00,510 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:02,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:35:02,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:35:02,391 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:35:06,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:07,355 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:35:07,356 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:35:07,356 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:35:07,357 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:35:07,774 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:35:09,775 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:35:12,107 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:15,265 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:35:17,022 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:35:17,266 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:17,392 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:35:17,392 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:35:22,446 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:27,447 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:31,327 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:35:31,328 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:35:31,329 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:35:31,329 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:35:31,784 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:35:32,122 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:35:32,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:35:32,392 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:35:32,530 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:33,785 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:35:37,531 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:42,532 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:45,270 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:35:47,120 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:35:47,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:35:47,392 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:35:48,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:53,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:56,647 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:35:56,648 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:35:56,648 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:35:56,648 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:35:56,794 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:35:59,398 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:35:59,795 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:36:02,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:36:02,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:36:02,392 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:36:04,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:09,430 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:14,431 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:15,267 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:36:17,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:36:17,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:36:17,392 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:36:19,475 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:21,983 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:36:21,984 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:36:21,984 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:36:21,986 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:36:22,804 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:36:23,804 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:36:24,717 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:29,718 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:32,121 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:36:32,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:36:32,392 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:36:35,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:40,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:45,268 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:36:46,269 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:46,955 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:36:46,956 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:36:46,956 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:36:46,957 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:36:47,398 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:36:47,409 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:36:47,409 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:36:47,813 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:36:49,813 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:36:51,686 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:36:56,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:01,688 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:02,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:37:02,399 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:37:02,399 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:37:07,467 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:12,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:12,573 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:37:12,575 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:37:12,575 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:37:12,577 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:37:12,822 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:37:15,269 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:37:15,823 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:37:17,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:37:17,402 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:37:17,403 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:37:18,446 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:23,448 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:28,449 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:32,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:37:32,399 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:37:32,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:37:34,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:37,234 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:37:37,235 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:37:37,236 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:37:37,236 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:37:37,833 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:37:39,834 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:37:39,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:44,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:45,270 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:37:47,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:37:47,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:37:47,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:37:50,532 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:37:55,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:00,534 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:02,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:38:02,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:38:02,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:38:03,331 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:38:03,333 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:38:03,333 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:38:03,333 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:38:03,844 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:38:05,845 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:38:06,102 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:11,103 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:15,271 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:38:16,272 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:17,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:38:17,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:38:17,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:38:21,468 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:26,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:27,859 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:38:27,860 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:38:27,861 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:38:27,861 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:38:28,854 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:38:29,855 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:38:31,614 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:32,127 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:38:32,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:38:32,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:38:37,530 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:42,531 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:45,272 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:38:47,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:38:47,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:38:47,400 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:38:48,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:50,420 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:38:50,421 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:38:50,421 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:38:50,422 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:38:50,863 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:38:53,865 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:38:54,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:38:59,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:02,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:39:02,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:39:02,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:39:04,492 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:09,493 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:14,364 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:39:14,365 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:39:14,365 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:39:14,366 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:39:14,873 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:39:15,120 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:15,273 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:39:17,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:39:17,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:39:17,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:39:17,874 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:39:20,531 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:25,532 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:30,533 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:32,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:39:32,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:39:32,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:39:36,485 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:37,316 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:39:37,317 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:39:37,317 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:39:37,318 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:39:37,881 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:39:39,882 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:39:42,074 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:45,274 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:39:47,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:39:47,275 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:47,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:39:47,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:39:52,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:39:57,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:02,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:40:02,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:40:02,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:40:02,452 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:02,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:40:02,925 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:40:02,925 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:40:02,926 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:40:03,891 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:40:05,892 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:40:07,672 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:12,672 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:15,275 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:40:17,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:40:17,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:40:17,402 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:40:18,495 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:23,496 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:25,164 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:40:25,165 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:40:25,166 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:40:25,166 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:40:25,900 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:40:27,901 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:40:28,912 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:32,128 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:40:32,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:40:32,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:40:34,464 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:39,465 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:44,466 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:45,276 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:40:47,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:40:47,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:40:47,401 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:40:48,489 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:40:48,490 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:40:48,490 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:40:48,491 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:40:48,910 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:40:50,224 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:40:51,911 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:40:55,224 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:00,225 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:02,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:41:02,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:41:02,402 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:41:05,444 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:10,445 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:10,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:41:10,762 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:41:10,763 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:41:10,763 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:41:10,920 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:41:13,921 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:41:15,277 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:41:16,278 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:17,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:41:17,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:41:17,402 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:41:21,461 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:26,462 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:31,463 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:32,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:41:32,402 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:41:32,402 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:41:34,555 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:41:34,557 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:41:34,557 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:41:34,557 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:41:34,929 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:41:36,678 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:37,930 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:41:41,679 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:45,278 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:41:47,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:41:47,279 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:47,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:41:47,402 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:41:52,478 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:57,479 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:41:57,582 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:41:57,583 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:41:57,583 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:41:57,583 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:41:57,937 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:41:59,938 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:42:02,129 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:42:02,403 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:42:02,403 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:42:02,524 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:07,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:12,527 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:15,279 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:42:17,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:42:17,402 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:42:17,402 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:42:18,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:23,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:23,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:42:23,803 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:42:23,804 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:42:23,804 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:42:23,947 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:42:25,947 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:42:28,848 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:32,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:42:32,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:42:32,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:42:34,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:39,540 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:44,541 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:45,280 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:42:47,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:42:47,403 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:42:47,404 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:42:50,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:42:54,080 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:42:54,082 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:42:54,082 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:42:54,083 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:42:54,958 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:42:55,959 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:42:56,187 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:01,188 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:02,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:43:02,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:43:02,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:43:06,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:11,515 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:15,281 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:43:17,130 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:43:17,282 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:17,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:43:17,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:43:18,280 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:43:18,282 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:43:18,282 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:43:18,282 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:43:18,968 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:43:19,968 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:43:23,267 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:28,268 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:32,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:43:32,403 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:43:32,404 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:43:33,524 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:38,525 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:43,526 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:45,202 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:43:45,204 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:43:45,204 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:43:45,204 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:43:45,282 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:43:45,979 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:43:47,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:43:47,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:43:47,404 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:43:47,979 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:43:49,446 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:54,447 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:43:59,448 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:02,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:44:02,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:44:02,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:44:04,467 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:05,604 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:44:05,605 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:44:05,605 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:44:05,605 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:44:05,988 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:44:07,989 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:44:10,323 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:15,283 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:44:16,284 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:17,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:44:17,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:44:17,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:44:21,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:26,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:29,426 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:44:29,427 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:44:29,427 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:44:29,428 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:44:29,997 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:44:31,998 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:44:32,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:44:32,276 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:32,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:44:32,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:44:37,516 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:42,517 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:45,284 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:44:47,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:44:47,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:44:47,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:44:48,476 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:53,009 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:44:53,010 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:44:53,010 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:44:53,011 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:44:53,707 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:44:54,007 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:44:56,008 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:44:58,708 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:02,132 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:45:02,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:45:02,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:45:04,472 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:09,473 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:14,475 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:14,542 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:45:14,543 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:45:14,543 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:45:14,543 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:45:15,015 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:45:15,286 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:45:16,015 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:45:17,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:45:17,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:45:17,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:45:20,432 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:25,433 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:30,434 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:32,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:45:32,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:45:32,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:45:35,453 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:40,323 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:45:40,324 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:45:40,324 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:45:40,324 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:45:41,025 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:45:41,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:42,025 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:45:45,287 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:45:46,287 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:47,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:45:47,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:45:47,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:45:51,499 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:45:56,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:00,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:46:00,778 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:46:00,778 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:46:00,779 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:46:01,032 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:46:02,033 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:46:02,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:46:02,227 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:02,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:46:02,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:46:07,440 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:12,441 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:15,287 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:46:17,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:46:17,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:46:17,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:46:17,444 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:22,446 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:24,682 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:46:24,683 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:46:24,683 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:46:24,684 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:46:25,041 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:46:27,042 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:46:27,492 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:32,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:46:32,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:46:32,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:46:33,445 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:38,446 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:43,447 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:45,288 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:46:47,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:46:47,405 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:46:47,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:46:48,464 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:51,289 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:46:51,290 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:46:51,291 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:46:51,291 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:46:52,051 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:46:53,757 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:46:55,052 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:46:58,758 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:02,133 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:47:02,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:47:02,406 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:47:04,447 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:09,448 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:14,449 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:15,289 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:47:17,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:47:17,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:47:17,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:47:18,560 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:47:18,561 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:47:18,562 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:47:18,562 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:47:19,061 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:47:20,302 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:21,062 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:47:25,304 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:30,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:32,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:47:32,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:47:32,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:47:35,449 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:40,450 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:41,802 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:47:41,803 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:47:41,803 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:47:41,804 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:47:42,071 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:47:45,072 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:47:45,290 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:47:46,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:47,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:47:47,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:47:47,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:47:51,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:47:56,459 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:01,460 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:02,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:48:02,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:48:02,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:48:04,131 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:48:04,132 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:48:04,132 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:48:04,133 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:48:05,079 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:48:06,876 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:07,080 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:48:11,877 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:15,291 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:48:17,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:48:17,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:17,406 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:48:17,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:48:22,518 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:24,214 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:48:24,215 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:48:24,215 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:48:24,216 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:48:25,087 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:48:27,088 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:48:27,686 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:32,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:48:32,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:48:32,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:48:33,462 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:38,463 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:43,464 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:45,293 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:48:47,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:48:47,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:48:47,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:48:49,451 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:48:50,374 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:48:50,376 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:48:50,376 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:48:50,376 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:48:51,098 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:48:53,099 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:48:55,194 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:00,195 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:02,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:49:02,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:49:02,407 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:49:05,486 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:10,487 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:14,179 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:49:14,180 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:49:14,181 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:49:14,181 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:49:15,108 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:49:15,294 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:49:16,295 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:17,109 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:49:17,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:49:17,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:49:17,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:49:21,528 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:26,529 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:31,530 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:32,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:49:32,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:49:32,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:49:37,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:49:37,310 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:49:37,310 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:49:37,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:37,311 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:49:38,118 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:49:41,119 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:49:43,093 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:45,295 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:49:47,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:49:47,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:49:47,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:49:48,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:53,508 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:57,064 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:49:57,065 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:49:57,065 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:49:57,066 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:49:57,125 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:49:58,852 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:49:59,126 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:50:02,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:50:02,407 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:50:02,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:50:04,455 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:09,456 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:14,457 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:15,296 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:50:17,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:50:17,408 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:50:17,408 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:50:19,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:21,695 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:50:21,696 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:50:21,697 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:50:21,697 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:50:22,136 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:50:25,137 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:50:25,445 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:30,446 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:32,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:50:32,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:50:32,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:50:35,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:40,539 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:45,297 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:50:45,666 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:50:45,667 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:50:45,667 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:50:45,668 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:45,668 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:50:46,145 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:50:47,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:50:47,410 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:50:47,411 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:50:49,146 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:50:51,549 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:50:56,550 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:01,551 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:02,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:51:02,412 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:51:02,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:51:07,508 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:10,197 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:51:10,198 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:51:10,199 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:51:10,199 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:51:11,154 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:51:13,156 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:51:13,194 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:15,299 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:51:17,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:51:17,410 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:51:17,411 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:51:18,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:23,506 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:28,507 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:32,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:51:32,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:51:32,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:51:33,545 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:35,439 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:51:35,441 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:51:35,441 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:51:35,441 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:51:36,164 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:51:39,165 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:51:39,238 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:44,239 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:45,299 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:51:47,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:51:47,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:51:47,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:51:49,511 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:54,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:51:58,362 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:51:58,363 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:51:58,363 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:51:58,364 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:51:59,173 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:51:59,881 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:01,174 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:52:02,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:52:02,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:52:02,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:52:05,469 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:10,470 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:15,301 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:52:16,302 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:17,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:52:17,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:52:17,411 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:52:21,192 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:52:21,193 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:52:21,193 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:52:21,194 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:52:21,959 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:22,184 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:52:23,185 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:52:26,960 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:31,961 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:32,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:52:32,412 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:52:32,413 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:52:37,455 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:42,456 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:45,302 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:52:47,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:52:47,376 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:52:47,377 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:52:47,377 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:52:47,378 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:52:47,417 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:52:47,417 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:52:48,168 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:48,195 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:52:51,197 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:52:53,169 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:52:58,170 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:02,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:53:02,411 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:53:02,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:53:03,475 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:08,476 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:10,534 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:53:10,535 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:53:10,536 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:53:10,536 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:53:11,205 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:53:13,206 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:53:14,326 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:15,358 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:53:17,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:53:17,412 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:53:17,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:53:19,528 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:24,529 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:29,530 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:32,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:53:32,412 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:53:32,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:53:35,481 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:35,880 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:53:35,881 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:53:35,882 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:53:35,882 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:53:36,215 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:53:39,217 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:53:40,657 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:45,304 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:53:46,305 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:47,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:53:47,412 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:53:47,412 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:53:51,537 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:53:56,538 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:01,539 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:02,137 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:54:02,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:54:02,163 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:54:02,164 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:54:02,166 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:54:02,227 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:54:02,654 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:54:02,654 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:54:05,228 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:54:06,959 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:11,960 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:15,305 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:54:17,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:54:17,306 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:17,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:54:17,609 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:54:22,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:24,414 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:54:24,416 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:54:24,416 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:54:24,416 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:54:25,236 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:54:27,237 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:54:28,184 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:32,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:54:32,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:54:32,609 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:54:33,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:38,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:43,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:45,307 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:54:47,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:54:47,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:54:47,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:54:49,706 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:49,982 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:54:49,983 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:54:49,984 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:54:49,984 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:54:50,245 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:54:53,247 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:54:54,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:54:59,924 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:02,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:55:02,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:55:02,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:55:05,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:10,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:12,651 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:55:12,652 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:55:12,653 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:55:12,653 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:55:13,254 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:55:15,255 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:55:15,308 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:55:16,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:17,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:55:17,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:55:17,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:55:21,686 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:26,687 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:31,688 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:32,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:55:32,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:55:32,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:55:37,683 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:37,821 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:55:37,822 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:55:37,822 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:55:37,823 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:55:38,263 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:55:41,264 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:55:43,585 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:45,309 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:55:47,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:55:47,609 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:55:47,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:55:48,711 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:53,712 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:55:58,712 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:01,291 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:56:01,292 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:56:01,292 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:56:01,293 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:56:02,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:56:02,274 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:56:02,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:56:02,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:56:03,274 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:56:03,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:08,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:13,749 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:15,310 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:56:17,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:56:17,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:56:17,610 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:56:19,650 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:24,651 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:26,283 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:56:26,285 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:56:26,285 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:56:26,286 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:56:26,286 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:56:29,287 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:56:30,069 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:32,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:56:32,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:56:32,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:56:35,656 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:40,657 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:45,311 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:56:46,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:47,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:56:47,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:56:47,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:56:48,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:56:48,745 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:56:48,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:56:48,745 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:56:49,296 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:56:51,297 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:56:51,520 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:56:56,521 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:01,522 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:02,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:57:02,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:57:02,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:57:06,668 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:11,669 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:15,311 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:57:17,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:57:17,312 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:17,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:57:17,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:57:17,793 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:57:17,794 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:57:17,795 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:57:17,795 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:57:18,308 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:57:21,310 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:57:22,605 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:27,606 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:32,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:57:32,607 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:32,610 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:57:32,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:57:37,653 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:42,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:57:42,514 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:57:42,514 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:57:42,515 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:57:43,029 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:43,318 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:57:45,312 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:57:45,318 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:57:47,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:57:47,611 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:57:47,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:57:48,733 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:53,734 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:57:58,735 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:02,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:58:02,611 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:58:02,611 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:58:04,727 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:07,958 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:58:07,960 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:58:07,960 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:58:07,960 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:58:08,327 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:58:09,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:11,328 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:58:14,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:15,314 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:58:17,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:58:17,611 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:58:17,612 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:58:20,667 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:25,669 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:30,670 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:32,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:58:32,611 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:58:32,612 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:58:33,203 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:58:33,204 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:58:33,204 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:58:33,205 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:58:33,337 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:58:35,338 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:58:35,755 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:40,756 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:45,314 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:58:46,315 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:47,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:58:47,611 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:58:47,612 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:58:51,675 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:55,061 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:58:55,062 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:58:55,063 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:58:55,063 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:58:55,349 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:58:56,841 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:58:57,351 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:59:01,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:02,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:59:02,612 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:59:02,612 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:59:07,676 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:12,678 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:15,315 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:59:17,094 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:59:17,095 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:59:17,096 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:59:17,096 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:59:17,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:59:17,360 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:59:17,641 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:59:17,642 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:59:17,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:19,361 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:59:22,930 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:27,931 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:32,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:59:32,629 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:59:32,629 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:59:33,727 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:38,728 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:41,660 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 09:59:41,661 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 09:59:41,661 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 09:59:41,662 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 09:59:42,374 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 09:59:43,375 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 09:59:44,208 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:45,316 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 09:59:47,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 09:59:47,629 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 09:59:47,629 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 09:59:49,742 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:54,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 09:59:59,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:02,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:00:02,629 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:00:02,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:00:05,019 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:00:05,021 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:00:05,021 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:00:05,021 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:05,022 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:00:05,386 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:00:07,387 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:00:10,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:15,317 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:00:16,318 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:17,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:00:17,629 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:00:17,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:00:21,757 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:26,758 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:30,325 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:00:30,326 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:00:30,327 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:00:30,327 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:00:30,395 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:00:32,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:32,141 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:00:32,629 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:00:32,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:00:33,396 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:00:37,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:42,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:45,318 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:00:47,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:00:47,629 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:00:47,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:00:48,759 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:53,760 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:00:58,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:00,898 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:01:00,899 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:01:00,899 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:01:00,900 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:01:01,407 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:01:02,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:01:02,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:01:02,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:01:03,409 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:01:04,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:09,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:14,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:15,319 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:01:17,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:01:17,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:01:17,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:01:20,675 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:25,676 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:30,226 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:01:30,227 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:01:30,228 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:01:30,228 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:01:30,426 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:01:30,758 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:31,427 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:01:32,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:01:32,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:01:32,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:01:36,676 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:41,677 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:45,320 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:01:47,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:01:47,321 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:47,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:01:47,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:01:51,292 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:01:51,293 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:01:51,293 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:01:51,294 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:01:51,434 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:01:53,077 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:01:53,435 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:01:58,078 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:02,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:02:02,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:02:02,630 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:02:03,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:08,724 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:13,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:15,321 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:02:17,031 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:02:17,032 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:02:17,032 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:02:17,033 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:02:17,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:02:17,445 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:02:17,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:02:17,766 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:02:18,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:19,446 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:02:23,812 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:28,813 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:32,142 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:02:32,718 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:02:32,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:02:34,811 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:38,193 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:02:38,195 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:02:38,195 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:02:38,195 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:02:38,453 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:02:40,105 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:41,455 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:02:45,107 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:45,322 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:02:47,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:02:47,718 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:02:47,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:02:50,761 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:55,762 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:02:59,977 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:02:59,978 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:02:59,979 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:02:59,979 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:03:00,467 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:03:01,468 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:03:01,500 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:02,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:03:02,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:03:02,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:03:06,838 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:11,839 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:15,323 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:03:17,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:03:17,324 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:17,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:03:17,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:03:22,803 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:25,933 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:03:25,935 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:03:25,935 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:03:25,935 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:03:26,483 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:03:27,484 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:03:28,452 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:32,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:03:32,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:03:32,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:03:33,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:38,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:43,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:45,324 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:03:47,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:03:47,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:03:47,719 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:03:48,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:03:48,502 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:03:48,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:03:48,502 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:03:49,307 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:49,498 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:03:51,499 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:03:54,308 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:03:59,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:02,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:04:02,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:04:02,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:04:04,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:09,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:11,624 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:04:11,626 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:04:11,626 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:04:11,626 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:04:12,512 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:04:13,513 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:04:15,325 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:04:15,325 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:17,143 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:04:17,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:04:17,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:04:20,784 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:25,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:30,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:32,144 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:04:32,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:04:32,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:04:35,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:38,307 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:04:38,308 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:04:38,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:04:38,309 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:04:38,529 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:04:39,529 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:04:40,907 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:45,326 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:04:46,326 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:47,144 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:04:47,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:04:47,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:04:51,877 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:04:56,878 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:01,879 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:02,144 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:05:02,444 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:05:02,446 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:05:02,446 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:05:02,446 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:05:02,543 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:05:02,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:05:02,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:05:05,545 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:05:07,309 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:12,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:15,327 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:05:17,144 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:05:17,328 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:17,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:05:17,721 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:05:22,769 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:25,718 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:05:25,719 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:05:25,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:05:25,720 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:05:26,554 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:05:27,555 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:05:28,240 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:32,144 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:05:32,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:05:32,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:05:33,788 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:38,789 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:43,790 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:45,328 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:05:47,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:05:47,666 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:05:47,667 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:05:47,668 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:05:47,668 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:05:47,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:05:47,720 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:05:48,562 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:05:49,435 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:49,562 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:05:54,436 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:05:59,437 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:02,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:06:02,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:06:02,721 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:06:04,800 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:09,801 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:13,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:06:13,444 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:06:13,445 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:06:13,445 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:06:13,572 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:06:15,001 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:15,328 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:06:15,574 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:06:17,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:06:17,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:06:17,721 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:06:20,778 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:25,779 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:30,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:32,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:06:32,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:06:32,721 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:06:35,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:37,686 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:06:37,687 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:06:37,688 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:06:37,688 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:06:38,583 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:06:39,584 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:06:41,636 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:45,329 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:06:47,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:06:47,331 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:47,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:06:47,721 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:06:52,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:06:57,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:01,244 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:07:01,246 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:07:01,246 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:07:01,246 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:07:01,592 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:07:02,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:07:02,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:07:02,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:07:03,594 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:07:03,813 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:08,814 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:13,815 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:15,331 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:07:17,145 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:07:17,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:07:17,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:07:19,759 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:24,760 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:29,185 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:07:29,186 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:07:29,187 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:07:29,187 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:07:29,605 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:07:30,007 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:31,606 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:07:32,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:07:32,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:07:32,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:07:35,831 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:40,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:45,332 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:07:46,333 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:47,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:07:47,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:07:47,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:07:51,812 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:07:54,959 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:07:54,960 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:07:54,961 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:07:54,961 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:07:55,615 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:07:57,616 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:07:57,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:02,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:08:02,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:08:02,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:08:02,763 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:07,764 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:12,765 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:15,332 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:08:17,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:08:17,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:08:17,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:08:17,809 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:22,140 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:08:22,141 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:08:22,142 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:08:22,142 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:08:22,625 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:08:22,930 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:23,625 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:08:27,932 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:32,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:08:32,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:08:32,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:08:33,844 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:38,846 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:43,847 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:45,333 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:08:47,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:08:47,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:08:47,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:08:48,196 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:08:48,198 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:08:48,198 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:08:48,198 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:08:48,638 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:08:49,134 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:49,638 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:08:54,135 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:08:59,136 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:02,146 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:09:02,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:09:02,722 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:09:04,830 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:09,831 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:12,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:09:12,444 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:09:12,445 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:09:12,445 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:09:12,647 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:09:13,647 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:09:15,224 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:15,334 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:09:17,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:09:17,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:09:17,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:09:20,768 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:25,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:30,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:32,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:09:32,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:09:32,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:09:35,841 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:39,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:09:39,430 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:09:39,430 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:09:39,430 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:09:39,660 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:09:41,202 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:41,661 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:09:45,336 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:09:46,337 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:47,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:09:47,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:09:47,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:09:51,843 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:09:56,844 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:01,845 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:02,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:10:02,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:10:02,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:10:05,054 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:10:05,055 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:10:05,055 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:10:05,056 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:10:05,671 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:10:07,230 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:07,672 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:10:12,231 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:15,337 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:10:17,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:10:17,338 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:17,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:10:17,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:10:22,754 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:27,754 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:29,773 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:10:29,775 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:10:29,775 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:10:29,775 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:10:30,681 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:10:31,682 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:10:32,147 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:10:32,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:10:32,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:10:32,805 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:37,806 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:42,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:45,338 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:10:47,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:10:47,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:10:47,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:10:47,856 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:52,857 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:57,858 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:10:58,470 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:10:58,471 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:10:58,472 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:10:58,472 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:10:58,699 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:10:59,702 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:11:02,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:11:02,723 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:11:02,723 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:11:03,831 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:08,832 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:13,833 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:15,338 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:11:17,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:11:17,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:11:17,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:11:19,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:22,649 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:11:22,654 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:11:22,655 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:11:22,655 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:11:22,710 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:11:24,711 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:11:25,428 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:30,429 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:32,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:11:32,724 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:11:32,725 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:11:36,031 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:41,032 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:45,339 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:11:46,340 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:47,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:11:47,681 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:11:47,682 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:11:47,682 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:11:47,683 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:11:47,720 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:11:47,724 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:11:47,725 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:11:50,721 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:11:51,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:11:56,443 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:01,444 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:02,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:12:02,724 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:12:02,725 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:12:06,857 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:10,756 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:12:10,757 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:12:10,757 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:12:10,758 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:12:11,731 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:12:12,679 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:14,733 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:12:15,340 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:12:17,148 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:12:17,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:17,726 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:12:17,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:12:22,844 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:27,845 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:32,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:12:32,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:12:32,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:12:32,856 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:37,857 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:37,993 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:12:37,994 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:12:37,995 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:12:37,995 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:12:38,742 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:12:40,742 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:12:43,814 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:45,341 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:12:47,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:12:47,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:12:47,725 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:12:48,827 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:53,828 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:12:58,829 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:02,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:13:02,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:13:02,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:13:03,983 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:13:03,984 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:13:03,984 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:13:03,985 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:03,985 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:13:04,751 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:13:06,752 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:13:09,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:14,359 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:15,342 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:13:17,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:13:17,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:13:17,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:13:19,819 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:24,820 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:29,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:13:29,402 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:13:29,403 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:13:29,403 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:13:29,760 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:13:30,159 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:32,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:13:32,725 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:13:32,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:13:32,761 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:13:36,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:41,139 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:45,343 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:13:46,344 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:47,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:13:47,726 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:13:47,726 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:13:51,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:54,945 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:13:54,946 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:13:54,946 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:13:54,947 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:13:55,771 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:13:57,740 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:13:58,772 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:14:02,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:14:02,727 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:14:02,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:14:02,775 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:07,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:12,777 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:15,344 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:14:17,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:14:17,726 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:14:17,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:14:17,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:19,964 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:14:19,966 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:14:19,966 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:14:19,966 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:14:20,781 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:14:22,781 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:14:23,735 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:28,736 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:32,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:14:32,726 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:14:32,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:14:33,770 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:38,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:43,772 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:45,345 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:14:46,045 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:14:46,047 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:14:46,047 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:14:46,047 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:14:46,790 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:14:47,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:14:47,726 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:14:47,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:14:48,779 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:48,791 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:14:53,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:14:58,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:02,150 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:15:02,726 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:15:02,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:15:04,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:09,777 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:11,471 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:15:11,473 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:15:11,473 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:15:11,473 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:15:11,800 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:15:14,801 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:15:15,346 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:15:15,346 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:17,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:15:17,727 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:15:17,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:15:20,849 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:25,850 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:30,851 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:32,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:15:32,727 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:15:32,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:15:36,771 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:37,274 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:15:37,275 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:15:37,275 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:15:37,276 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:15:37,812 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:15:40,814 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:15:42,138 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:45,347 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:15:47,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:15:47,348 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:47,727 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:15:47,727 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:15:52,845 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:15:57,846 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:01,901 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:16:01,902 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:16:01,902 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:16:01,902 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:16:02,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:16:02,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:16:02,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:16:02,822 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:16:03,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:04,823 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:16:08,787 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:13,788 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:15,348 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:16:17,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:16:17,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:16:17,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:16:18,871 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:23,872 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:26,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:16:26,799 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:16:26,799 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:16:26,800 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:16:26,831 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:16:28,832 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:16:29,614 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:32,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:16:32,742 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:16:32,742 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:16:34,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:39,777 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:44,778 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:45,349 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:16:47,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:16:47,742 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:16:47,742 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:16:48,822 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:16:48,823 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:16:48,823 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:16:48,823 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:16:48,840 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:16:50,644 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:16:50,841 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:16:55,645 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:00,646 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:02,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:17:02,742 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:17:02,743 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:17:05,826 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:10,827 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:12,898 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:17:12,899 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:17:12,899 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:17:12,900 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:17:13,850 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:17:14,851 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:17:15,349 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:17:16,351 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:17,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:17:17,742 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:17:17,743 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:17:21,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:26,787 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:31,788 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:32,151 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:17:32,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:17:32,743 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:17:36,496 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:17:36,497 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:17:36,497 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:17:36,498 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:17:36,859 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:17:37,298 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:38,860 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:17:42,299 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:45,350 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:17:47,152 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:17:47,351 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:47,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:17:47,743 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:17:52,794 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:17:57,795 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:00,349 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:18:00,350 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:18:00,350 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:18:00,351 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:18:00,868 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:18:02,152 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:18:02,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:18:02,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:18:02,869 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:18:03,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:08,775 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:13,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:15,351 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:18:17,152 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:18:17,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:18:17,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:18:18,822 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:23,823 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:24,303 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:18:24,304 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:18:24,304 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:18:24,305 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:18:24,877 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:18:26,878 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:18:29,106 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:32,152 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:18:32,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:18:32,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:18:34,841 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:39,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:44,844 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:45,352 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:18:47,152 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:18:47,743 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:18:47,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:18:47,903 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:18:47,904 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:18:47,904 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:18:47,904 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:18:48,888 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:18:50,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:18:50,889 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:18:55,721 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:00,722 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:02,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:19:02,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:19:02,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:19:05,864 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:10,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:11,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:19:11,775 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:19:11,775 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:19:11,776 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:19:11,897 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:19:14,899 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:19:15,353 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:19:16,354 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:17,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:19:17,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:19:17,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:19:21,865 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:26,866 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:31,867 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:32,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:19:32,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:19:32,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:19:36,393 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:19:36,395 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:19:36,395 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:19:36,395 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:19:36,910 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:19:37,177 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:38,911 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:19:42,178 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:45,354 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:19:47,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:19:47,355 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:47,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:19:47,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:19:52,848 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:19:57,849 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:00,902 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:20:00,904 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:20:00,904 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:20:00,904 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:20:00,919 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:20:02,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:20:02,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:20:02,744 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:20:02,920 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:20:03,817 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:08,818 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:13,819 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:15,359 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:20:17,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:20:17,744 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:20:17,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:20:19,783 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:23,839 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:20:23,840 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:20:23,841 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:20:23,841 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:20:23,928 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:20:25,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:26,929 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:20:30,631 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:32,153 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:20:32,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:20:32,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:20:35,833 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:40,834 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:45,356 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:20:46,357 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:46,872 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:20:46,873 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:20:46,874 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:20:46,876 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:20:46,937 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:20:47,379 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:20:47,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:20:47,750 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:20:48,938 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:20:51,821 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:20:56,822 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:01,823 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:02,332 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:21:02,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:21:02,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:21:06,842 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:11,843 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:13,261 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:21:13,262 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:21:13,263 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:21:13,263 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:21:13,949 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:21:14,950 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:21:15,357 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:21:17,332 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:21:17,358 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:17,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:21:17,745 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:21:22,834 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:27,835 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:32,332 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:21:32,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:21:32,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:21:32,868 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:37,404 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:21:37,405 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:21:37,405 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:21:37,408 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:21:37,958 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:21:38,270 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:38,959 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:21:43,271 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:45,358 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:21:47,332 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:21:47,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:21:47,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:21:48,882 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:53,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:21:58,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:01,505 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:22:01,506 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:22:01,506 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:22:01,507 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:22:01,968 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:22:02,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:22:02,745 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:22:02,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:22:04,867 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:04,970 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:22:09,868 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:14,869 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:15,359 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:22:17,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:22:17,746 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:22:17,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:22:20,815 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:25,816 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:26,384 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:22:26,385 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:22:26,386 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:22:26,386 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:22:26,978 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:22:28,979 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:22:31,190 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:32,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:22:32,746 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:22:32,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:22:36,860 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:41,861 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:45,359 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:22:47,360 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:47,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:22:47,746 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:22:47,746 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:22:52,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:22:53,042 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:22:53,043 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:22:53,043 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:22:53,044 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:22:53,990 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:22:54,989 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:22:57,830 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:02,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:23:02,746 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:23:02,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:23:02,855 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:07,856 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:12,857 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:15,360 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:23:17,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:23:17,746 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:23:17,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:23:17,870 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:18,306 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:23:18,307 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:23:18,308 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:23:18,308 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:23:18,999 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:23:21,000 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:23:23,270 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:28,271 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:32,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:23:32,746 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:23:32,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:23:33,825 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:38,826 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:41,868 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:23:41,869 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:23:41,869 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:23:41,870 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:23:42,008 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:23:44,635 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:45,009 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:23:45,361 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:23:47,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:23:47,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:23:47,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:23:49,819 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:54,820 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:23:59,821 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:02,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:24:02,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:24:02,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:24:05,812 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:06,104 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:24:06,105 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:24:06,105 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:24:06,106 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:24:07,018 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:24:09,019 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:24:10,925 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:15,362 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:24:16,363 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:17,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:24:17,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:24:17,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:24:21,814 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:26,815 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:30,705 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:24:30,706 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:24:30,706 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:24:30,707 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:24:31,028 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:24:31,956 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:32,389 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:24:32,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:24:32,747 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:24:33,029 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:24:37,788 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:42,789 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:45,363 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:24:47,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:24:47,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:24:47,748 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:24:47,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:52,839 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:24:54,851 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:24:54,852 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:24:54,853 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:24:54,853 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:24:55,037 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:24:57,038 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:24:58,630 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:02,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:25:02,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:25:02,748 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:25:03,797 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:08,798 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:13,799 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:15,364 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:25:17,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:25:17,747 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:25:17,748 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:25:18,872 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:23,282 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:25:23,283 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:25:23,283 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:25:23,284 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:25:24,051 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:25:24,095 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:25,052 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:25:29,096 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:32,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:25:32,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:25:32,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:25:34,942 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:39,944 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:44,945 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:45,365 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:25:47,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:25:47,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:25:47,748 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:25:48,570 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:25:48,572 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:25:48,572 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:25:48,572 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:25:49,061 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:25:50,512 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:25:51,062 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:25:55,513 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:00,514 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:02,390 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:26:02,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:26:02,748 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:26:05,869 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:10,870 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:14,272 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:26:14,273 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:26:14,273 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:26:14,274 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:26:15,071 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:26:15,366 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:26:16,367 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:17,072 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:26:17,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:26:17,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:26:17,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:26:21,972 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:26,973 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:31,974 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:32,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:26:32,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:26:32,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:26:37,824 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:40,162 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:26:40,164 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:26:40,164 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:26:40,164 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:26:41,082 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:26:42,990 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:43,083 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:26:45,367 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:26:47,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:26:47,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:26:47,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:26:48,870 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:53,872 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:26:58,873 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:02,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:27:02,748 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:27:02,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:27:02,907 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:27:02,908 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:27:02,908 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:27:02,909 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:27:03,092 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:27:04,718 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:05,093 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:27:09,719 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:14,720 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:15,368 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:27:17,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:27:17,749 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:27:17,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:27:19,785 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:24,786 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:26,672 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:27:26,673 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:27:26,673 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:27:26,673 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:27:27,102 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:27:29,103 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:27:30,472 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:32,391 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:27:32,749 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:27:32,749 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:27:35,870 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:40,871 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:45,369 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:27:46,370 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:47,392 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:27:47,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:27:47,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:27:50,577 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:27:50,579 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:27:50,579 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:27:50,579 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:27:51,112 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:27:51,378 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:27:53,113 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:27:56,380 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:01,381 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:02,396 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:28:02,750 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:28:02,750 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:28:06,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:11,808 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:15,370 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:28:15,841 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:28:15,843 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:28:15,843 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:28:15,844 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:28:16,122 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:28:17,123 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:28:17,395 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:28:17,636 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:17,750 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:28:17,750 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:28:22,869 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:27,871 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:32,402 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:28:32,750 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:28:32,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:28:32,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:37,353 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:28:37,355 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:28:37,355 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:28:37,355 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:28:38,132 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:28:38,197 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:39,132 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:28:43,198 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:45,371 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:28:47,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:28:47,750 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:28:47,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:28:48,806 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:53,807 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:28:58,808 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:02,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:29:02,750 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:29:02,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:29:02,923 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:29:02,925 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:29:02,925 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:29:02,925 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:29:03,141 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:29:04,775 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:05,142 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:29:09,776 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:14,777 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:15,376 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:29:17,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:29:17,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:29:17,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:29:19,877 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:24,438 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:29:24,439 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:29:24,439 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:29:24,440 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:29:25,149 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:29:25,310 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:27,150 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:29:30,311 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:32,400 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:29:32,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:29:32,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:29:35,882 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:40,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:45,373 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:29:46,374 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:47,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:29:47,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:29:47,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:29:50,097 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:29:50,098 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:29:50,098 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:29:50,101 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:29:50,159 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:29:51,160 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:29:51,618 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:29:56,619 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:01,620 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:02,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:30:02,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:30:02,752 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:30:06,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:11,841 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:14,601 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:30:14,602 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:30:14,602 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:30:14,603 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:30:15,170 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:30:15,374 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:30:17,171 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:30:17,388 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:17,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:30:17,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:30:17,751 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:30:22,774 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:27,775 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:32,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:30:32,751 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:30:32,752 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:30:32,897 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:34,471 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:30:34,472 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:30:34,472 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:30:34,474 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:30:35,177 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:30:37,178 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:30:38,089 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:43,090 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:45,375 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:30:47,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:30:47,752 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:30:47,752 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:30:48,838 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:53,839 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:58,840 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:30:59,149 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:30:59,150 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:30:59,150 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:30:59,151 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:30:59,186 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json
2024-04-12 10:31:01,187 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/output.log
2024-04-12 10:31:02,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:31:02,752 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:31:02,753 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:31:03,883 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:31:08,884 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:31:13,885 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:31:15,375 DEBUG   SenderThread:485 [sender.py:send():379] send: stats
2024-04-12 10:31:17,401 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-12 10:31:17,752 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: stop_status
2024-04-12 10:31:17,752 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: stop_status
2024-04-12 10:31:19,780 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: status_report
2024-04-12 10:31:22,501 DEBUG   HandlerThread:485 [handler.py:handle_request():146] handle_request: partial_history
2024-04-12 10:31:22,502 DEBUG   SenderThread:485 [sender.py:send():379] send: history
2024-04-12 10:31:22,502 DEBUG   SenderThread:485 [sender.py:send_request():406] send_request: summary_record
2024-04-12 10:31:22,503 INFO    SenderThread:485 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-12 10:31:23,197 INFO    Thread-12 :485 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240412_074044-widovcn3/files/wandb-summary.json