2024-02-01 17:38:28,445 INFO StreamThr :237521 [internal.py:wandb_internal():86] W&B internal server running at pid: 237521, started at: 2024-02-01 17:38:28.443368 2024-02-01 17:38:28,446 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status 2024-02-01 17:38:28,454 INFO WriterThread:237521 [datastore.py:open_for_write():85] open: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb 2024-02-01 17:38:28,455 DEBUG SenderThread:237521 [sender.py:send():382] send: header 2024-02-01 17:38:28,476 DEBUG SenderThread:237521 [sender.py:send():382] send: run 2024-02-01 17:38:28,713 INFO SenderThread:237521 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files 2024-02-01 17:38:28,713 INFO SenderThread:237521 [sender.py:_start_run_threads():1136] run started: py26nu6m with start time 1706809108.451874 2024-02-01 17:38:28,720 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: check_version 2024-02-01 17:38:28,720 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: check_version 2024-02-01 17:38:28,774 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: run_start 2024-02-01 17:38:28,833 DEBUG HandlerThread:237521 [system_info.py:__init__():32] System info init 2024-02-01 17:38:28,833 DEBUG HandlerThread:237521 [system_info.py:__init__():47] System info init done 2024-02-01 17:38:28,833 INFO HandlerThread:237521 [system_monitor.py:start():194] Starting system monitor 2024-02-01 17:38:28,833 INFO SystemMonitor:237521 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-01 17:38:28,833 INFO HandlerThread:237521 [system_monitor.py:probe():214] Collecting system info 2024-02-01 17:38:28,834 INFO SystemMonitor:237521 [interfaces.py:start():190] Started cpu monitoring 2024-02-01 17:38:28,834 INFO SystemMonitor:237521 [interfaces.py:start():190] Started disk monitoring 2024-02-01 17:38:28,834 INFO SystemMonitor:237521 [interfaces.py:start():190] Started gpu monitoring 2024-02-01 17:38:28,836 INFO SystemMonitor:237521 [interfaces.py:start():190] Started memory monitoring 2024-02-01 17:38:28,836 INFO SystemMonitor:237521 [interfaces.py:start():190] Started network monitoring 2024-02-01 17:38:28,880 DEBUG HandlerThread:237521 [system_info.py:probe():196] Probing system 2024-02-01 17:38:28,883 DEBUG HandlerThread:237521 [system_info.py:_probe_git():181] Probing git 2024-02-01 17:38:28,908 DEBUG HandlerThread:237521 [system_info.py:_probe_git():189] Probing git done 2024-02-01 17:38:28,908 DEBUG HandlerThread:237521 [system_info.py:probe():244] Probing system done 2024-02-01 17:38:28,908 DEBUG HandlerThread:237521 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.5', 'heartbeatAt': '2024-02-01T17:38:28.880477', 'startedAt': '2024-02-01T17:38:28.419493', 'docker': None, 'cuda': None, 'args': ('config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-ssft/run_sft.py', 'codePathLocal': 'run_sft.py', 'codePath': 'run_sft.py', 'git': {'remote': 'https://huggingface.co./sanchit-gandhi/distil-zephyr-1.5b-ssft', 'commit': '79a4ae874a71e67016ded927e7d23351e5c7dab8'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-ssft', 'host': 'ip-26-0-165-24', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/venv/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2731.1230833333334, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.237, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3593.987, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.474, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3587.972, 'min': 0.0, 'max': 0.0}, {'current': 3597.373, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2899.882, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.404, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.582, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 57.44935989379883}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.9855346679688}} 2024-02-01 17:38:28,908 INFO HandlerThread:237521 [system_monitor.py:probe():224] Finished collecting system info 2024-02-01 17:38:28,908 INFO HandlerThread:237521 [system_monitor.py:probe():227] Publishing system info 2024-02-01 17:38:28,908 DEBUG HandlerThread:237521 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2024-02-01 17:38:28,910 DEBUG HandlerThread:237521 [system_info.py:_save_pip():68] Saving pip packages done 2024-02-01 17:38:28,910 DEBUG HandlerThread:237521 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2024-02-01 17:38:29,716 INFO Thread-12 :237521 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/conda-environment.yaml 2024-02-01 17:38:29,716 INFO Thread-12 :237521 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/requirements.txt 2024-02-01 17:38:34,420 DEBUG HandlerThread:237521 [system_info.py:_save_conda():87] Saving conda packages done 2024-02-01 17:38:34,423 INFO HandlerThread:237521 [system_monitor.py:probe():229] Finished publishing system info 2024-02-01 17:38:34,457 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:38:34,457 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: keepalive 2024-02-01 17:38:34,458 DEBUG SenderThread:237521 [sender.py:send():382] send: files 2024-02-01 17:38:34,458 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now 2024-02-01 17:38:34,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:38:34,465 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:38:34,466 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:38:34,553 DEBUG SenderThread:237521 [sender.py:send():382] send: telemetry 2024-02-01 17:38:34,554 DEBUG SenderThread:237521 [sender.py:send():382] send: config 2024-02-01 17:38:34,555 DEBUG SenderThread:237521 [sender.py:send():382] send: metric 2024-02-01 17:38:34,555 DEBUG SenderThread:237521 [sender.py:send():382] send: telemetry 2024-02-01 17:38:34,555 DEBUG SenderThread:237521 [sender.py:send():382] send: metric 2024-02-01 17:38:34,555 WARNING SenderThread:237521 [sender.py:send_metric():1343] Seen metric with glob (shouldn't happen) 2024-02-01 17:38:34,555 DEBUG SenderThread:237521 [sender.py:send():382] send: telemetry 2024-02-01 17:38:34,721 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/conda-environment.yaml 2024-02-01 17:38:34,722 INFO Thread-12 :237521 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-metadata.json 2024-02-01 17:38:34,722 INFO Thread-12 :237521 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:34,796 INFO wandb-upload_0:237521 [upload_job.py:push():131] Uploaded file /tmp/tmpfs5f2n7fwandb/421ry27q-wandb-metadata.json 2024-02-01 17:38:36,724 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:39,559 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:38:42,110 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:38:42,112 DEBUG SenderThread:237521 [sender.py:send():382] send: metric 2024-02-01 17:38:42,112 DEBUG SenderThread:237521 [sender.py:send():382] send: metric 2024-02-01 17:38:42,112 DEBUG SenderThread:237521 [sender.py:send():382] send: metric 2024-02-01 17:38:42,112 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:38:42,113 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:38:42,116 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:38:42,732 INFO Thread-12 :237521 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:38:44,735 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:44,770 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:38:45,673 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:38:45,674 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:38:45,674 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:38:45,676 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:38:45,737 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:38:46,738 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:48,741 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:49,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:38:49,465 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:38:49,467 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:38:50,190 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:38:50,230 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:38:50,232 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:38:50,232 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:38:50,234 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:38:50,745 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:38:50,745 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:52,747 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:54,715 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:38:54,716 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:38:54,716 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:38:54,718 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:38:54,751 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:38:54,751 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:55,617 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:38:56,753 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:58,756 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:38:59,239 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:38:59,240 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:38:59,240 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:38:59,242 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:38:59,758 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:00,760 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:01,049 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:01,762 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/config.yaml 2024-02-01 17:39:02,763 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:03,754 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:03,755 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:03,755 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:03,757 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:03,766 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:04,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:39:04,466 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:39:04,466 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:39:04,767 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:06,478 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:06,770 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:08,293 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:08,294 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:08,295 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:08,297 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:08,773 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:08,774 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:10,776 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:11,923 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:12,779 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:12,828 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:12,830 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:12,830 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:12,832 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:13,781 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:14,782 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:16,785 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:17,363 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:17,365 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:17,366 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:17,367 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:17,368 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:17,788 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:18,789 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:19,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:39:19,466 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:39:19,467 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:39:20,792 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:21,909 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:21,911 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:21,911 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:21,913 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:22,795 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:22,796 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:22,815 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:24,798 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:26,448 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:26,450 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:26,450 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:26,452 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:26,802 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:26,802 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:28,269 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:28,805 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:28,836 DEBUG SystemMonitor:237521 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-01 17:39:28,850 DEBUG SenderThread:237521 [sender.py:send():382] send: stats 2024-02-01 17:39:30,807 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:31,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:31,003 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:31,003 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:31,005 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:31,810 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:32,811 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:33,729 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:34,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:39:34,465 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:39:34,468 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:39:34,814 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:35,548 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:35,550 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:35,550 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:35,552 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:35,816 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:36,817 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:38,820 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:39,188 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:40,104 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:40,105 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:40,105 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:40,108 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:40,824 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:40,824 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:42,826 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:44,651 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:44,652 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:44,653 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:44,653 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:44,655 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:44,830 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:44,830 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:46,833 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:48,835 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:49,211 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:49,212 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:49,212 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:49,214 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:49,466 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:39:49,466 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:39:49,467 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:39:49,838 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:50,121 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:50,839 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:52,842 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:53,762 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:53,763 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:53,763 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:53,765 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:53,844 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:54,845 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:55,580 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:39:56,848 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:39:58,314 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:39:58,316 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:39:58,316 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:39:58,318 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:39:58,842 DEBUG SenderThread:237521 [sender.py:send():382] send: stats 2024-02-01 17:39:58,852 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:39:58,852 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:00,854 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:01,035 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:02,853 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:02,855 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:02,855 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:02,857 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:02,858 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:02,859 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:04,465 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:40:04,466 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:40:04,467 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:40:04,861 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:06,498 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:06,864 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:07,408 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:07,409 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:07,409 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:07,411 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:07,866 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:08,867 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:10,870 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:11,953 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:11,954 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:11,955 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:11,956 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:11,958 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:12,874 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:12,874 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:14,876 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:16,879 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:17,215 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:18,882 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:19,958 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:40:19,959 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:40:19,959 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:40:22,278 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:22,888 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:24,892 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:26,895 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:27,568 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:27,569 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:27,571 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:27,571 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:27,573 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:27,897 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:28,844 DEBUG SenderThread:237521 [sender.py:send():382] send: stats 2024-02-01 17:40:28,898 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:30,901 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:32,115 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:32,117 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:32,117 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:32,119 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:32,904 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:32,904 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:33,026 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:34,907 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:34,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:40:34,920 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:40:34,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:40:36,665 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:36,667 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:36,667 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:36,669 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:36,910 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:36,911 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:38,487 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:38,913 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:40,915 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:41,219 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:41,220 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:41,221 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:41,223 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:41,917 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:42,919 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:43,949 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:44,922 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:45,773 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:45,775 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:45,776 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:45,778 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:45,924 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:46,925 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:48,927 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:49,410 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:49,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:40:49,920 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:40:49,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:40:50,362 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:50,930 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:51,403 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:51,403 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:51,405 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:51,932 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:52,934 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:54,873 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:40:54,874 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:54,876 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:54,876 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:54,878 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:54,937 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:40:54,937 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:56,940 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:58,846 DEBUG SenderThread:237521 [sender.py:send():382] send: stats 2024-02-01 17:40:58,942 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:40:59,420 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:40:59,422 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:40:59,422 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:40:59,424 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:40:59,944 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:00,330 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:41:00,946 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:02,948 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:03,975 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:41:03,976 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:41:03,976 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:41:03,978 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:41:04,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:41:04,920 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:41:04,952 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:04,952 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:04,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:41:05,797 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:41:06,955 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:08,527 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:41:08,529 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:41:08,529 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:41:08,531 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:41:08,958 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:08,959 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:10,961 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:11,264 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:41:12,964 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:13,085 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:41:13,086 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:41:13,087 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:41:13,089 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:41:13,966 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:14,967 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:16,736 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:41:16,969 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:17,649 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:41:17,650 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:41:17,651 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:41:17,653 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:41:17,972 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:18,973 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:19,920 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:41:19,920 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:41:19,957 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:41:20,976 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:22,234 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:41:22,235 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:41:22,236 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:41:22,236 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:41:22,238 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:41:22,979 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:22,979 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:24,981 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:26,803 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:41:26,804 DEBUG SenderThread:237521 [sender.py:send():382] send: history 2024-02-01 17:41:26,805 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:41:26,806 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:41:26,985 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json 2024-02-01 17:41:26,985 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log 2024-02-01 17:41:27,718 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:41:28,848 DEBUG SenderThread:237521 [sender.py:send():382] send: stats 2024-02-01 17:41:28,987 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log