2024-02-01 17:58:50,082 INFO StreamThr :239784 [internal.py:wandb_internal():86] W&B internal server running at pid: 239784, started at: 2024-02-01 17:58:50.080918 2024-02-01 17:58:50,083 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status 2024-02-01 17:58:50,087 INFO WriterThread:239784 [datastore.py:open_for_write():85] open: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/run-i93q0p12.wandb 2024-02-01 17:58:50,088 DEBUG SenderThread:239784 [sender.py:send():382] send: header 2024-02-01 17:58:50,109 DEBUG SenderThread:239784 [sender.py:send():382] send: run 2024-02-01 17:58:50,401 INFO SenderThread:239784 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files 2024-02-01 17:58:50,401 INFO SenderThread:239784 [sender.py:_start_run_threads():1136] run started: i93q0p12 with start time 1706810330.083967 2024-02-01 17:58:50,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: check_version 2024-02-01 17:58:50,409 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: check_version 2024-02-01 17:58:50,463 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: run_start 2024-02-01 17:58:50,558 DEBUG HandlerThread:239784 [system_info.py:__init__():32] System info init 2024-02-01 17:58:50,558 DEBUG HandlerThread:239784 [system_info.py:__init__():47] System info init done 2024-02-01 17:58:50,558 INFO HandlerThread:239784 [system_monitor.py:start():194] Starting system monitor 2024-02-01 17:58:50,558 INFO SystemMonitor:239784 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-01 17:58:50,559 INFO HandlerThread:239784 [system_monitor.py:probe():214] Collecting system info 2024-02-01 17:58:50,559 INFO SystemMonitor:239784 [interfaces.py:start():190] Started cpu monitoring 2024-02-01 17:58:50,559 INFO SystemMonitor:239784 [interfaces.py:start():190] Started disk monitoring 2024-02-01 17:58:50,560 INFO SystemMonitor:239784 [interfaces.py:start():190] Started gpu monitoring 2024-02-01 17:58:50,560 INFO SystemMonitor:239784 [interfaces.py:start():190] Started memory monitoring 2024-02-01 17:58:50,561 INFO SystemMonitor:239784 [interfaces.py:start():190] Started network monitoring 2024-02-01 17:58:50,615 DEBUG HandlerThread:239784 [system_info.py:probe():196] Probing system 2024-02-01 17:58:50,618 DEBUG HandlerThread:239784 [system_info.py:_probe_git():181] Probing git 2024-02-01 17:58:50,642 DEBUG HandlerThread:239784 [system_info.py:_probe_git():189] Probing git done 2024-02-01 17:58:50,642 DEBUG HandlerThread:239784 [system_info.py:probe():244] Probing system done 2024-02-01 17:58:50,642 DEBUG HandlerThread:239784 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.5', 'heartbeatAt': '2024-02-01T17:58:50.615903', 'startedAt': '2024-02-01T17:58:50.064651', 'docker': None, 'cuda': None, 'args': ('config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-ssft/run_sft.py', 'codePathLocal': 'run_sft.py', 'codePath': 'run_sft.py', 'git': {'remote': 'https://huggingface.co./sanchit-gandhi/distil-zephyr-1.5b-ssft', 'commit': '79a4ae874a71e67016ded927e7d23351e5c7dab8'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-ssft', 'host': 'ip-26-0-165-24', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/venv/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2728.6971041666666, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3589.018, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3587.597, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3596.958, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.83, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3595.048, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3591.991, 'min': 0.0, 'max': 0.0}, {'current': 3598.381, 'min': 0.0, 'max': 0.0}, {'current': 3598.099, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 57.44965744018555}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.9855346679688}} 2024-02-01 17:58:50,642 INFO HandlerThread:239784 [system_monitor.py:probe():224] Finished collecting system info 2024-02-01 17:58:50,642 INFO HandlerThread:239784 [system_monitor.py:probe():227] Publishing system info 2024-02-01 17:58:50,642 DEBUG HandlerThread:239784 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2024-02-01 17:58:50,644 DEBUG HandlerThread:239784 [system_info.py:_save_pip():68] Saving pip packages done 2024-02-01 17:58:50,644 DEBUG HandlerThread:239784 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2024-02-01 17:58:51,404 INFO Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/requirements.txt 2024-02-01 17:58:51,404 INFO Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/conda-environment.yaml 2024-02-01 17:58:55,310 DEBUG HandlerThread:239784 [system_info.py:_save_conda():87] Saving conda packages done 2024-02-01 17:58:55,312 INFO HandlerThread:239784 [system_monitor.py:probe():229] Finished publishing system info 2024-02-01 17:58:55,339 DEBUG SenderThread:239784 [sender.py:send():382] send: files 2024-02-01 17:58:55,339 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now 2024-02-01 17:58:55,339 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:58:55,347 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:58:55,347 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:58:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:58:55,408 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/conda-environment.yaml 2024-02-01 17:58:55,409 INFO Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-metadata.json 2024-02-01 17:58:55,462 DEBUG SenderThread:239784 [sender.py:send():382] send: telemetry 2024-02-01 17:58:55,463 DEBUG SenderThread:239784 [sender.py:send():382] send: config 2024-02-01 17:58:55,463 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 17:58:55,463 DEBUG SenderThread:239784 [sender.py:send():382] send: telemetry 2024-02-01 17:58:55,463 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 17:58:55,464 WARNING SenderThread:239784 [sender.py:send_metric():1343] Seen metric with glob (shouldn't happen) 2024-02-01 17:58:55,464 DEBUG SenderThread:239784 [sender.py:send():382] send: telemetry 2024-02-01 17:58:55,631 INFO wandb-upload_0:239784 [upload_job.py:push():131] Uploaded file /tmp/tmpr1qr0ps0wandb/r53qkhjh-wandb-metadata.json 2024-02-01 17:58:56,411 INFO Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:58:58,413 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:00,471 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:05,472 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:06,418 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:59:06,420 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 17:59:06,420 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 17:59:06,421 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 17:59:06,421 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 17:59:06,421 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:59:06,424 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:59:06,424 INFO Thread-12 :239784 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 17:59:08,427 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:10,347 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:59:10,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:59:10,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:59:10,493 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:12,432 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:15,576 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:16,437 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:18,439 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:20,788 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:21,444 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/config.yaml 2024-02-01 17:59:22,446 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:25,347 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:59:25,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:59:25,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:59:26,451 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:26,471 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:30,456 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:31,603 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:34,461 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:36,464 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:37,209 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:38,807 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 17:59:38,808 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 17:59:38,808 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 17:59:38,812 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 17:59:39,469 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 17:59:40,347 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:59:40,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:59:40,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:59:40,470 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:42,419 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:44,475 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:48,018 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:48,480 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:50,561 DEBUG SystemMonitor:239784 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-01 17:59:50,564 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 17:59:52,485 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:53,221 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 17:59:54,488 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:55,347 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 17:59:55,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 17:59:55,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 17:59:57,492 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 17:59:58,833 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:01,497 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:04,040 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:05,502 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:09,507 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:09,647 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:10,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:00:10,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:00:10,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:00:11,509 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:14,847 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:14,849 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:00:14,850 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:00:14,851 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:00:14,853 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:00:15,516 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:15,516 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:00:16,517 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:19,521 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:20,459 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:20,566 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:00:23,526 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:25,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:00:25,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:00:25,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:00:25,478 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:27,531 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:29,534 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:31,279 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:33,539 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:36,505 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:37,544 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:40,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:00:40,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:00:40,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:00:41,549 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:42,498 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:45,554 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:47,556 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:48,306 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:50,568 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:00:50,912 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:00:50,913 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:00:50,914 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:00:50,915 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:00:51,562 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:51,563 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:00:52,564 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:53,925 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:55,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:00:55,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:00:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:00:55,567 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:00:59,109 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:00:59,572 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:03,578 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:04,714 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:05,580 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:09,585 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:09,913 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:10,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:01:10,348 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:01:10,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:01:13,590 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:15,510 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:17,595 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:19,526 WARNING FileStreamThread:239784 [file_stream.py:request_with_retry():668] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/sanchit-gandhi/huggingface/i93q0p12/file_stream. func: functools.partial(>, timeout=180), args: ('https://api.wandb.ai/files/sanchit-gandhi/huggingface/i93q0p12/file_stream',), kwargs: {'json': {'files': {'wandb-summary.json': {'offset': 0, 'content': ['{"train/loss": 7.6246, "train/learning_rate": 4.3795620437956196e-05, "train/epoch": 0.07, "train/global_step": 20, "_timestamp": 1706810414.8489923, "_runtime": 84.76502537727356, "_step": 2}']}, 'wandb-history.jsonl': {'offset': 2, 'content': ['{"train/loss": 7.6246, "train/learning_rate": 4.3795620437956196e-05, "train/epoch": 0.07, "train/global_step": 20, "_timestamp": 1706810414.8489923, "_runtime": 84.76502537727356, "_step": 2}']}, 'output.log': {'offset': 5, 'content': ["2024-02-01T18:00:16.385360 {'loss': 7.6246, 'learning_rate': 4.3795620437956196e-05, 'epoch': 0.07}\n"]}}, 'dropped': 0}} 2024-02-01 18:01:20,570 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:01:20,570 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:21,600 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:23,603 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:25,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:01:25,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:01:25,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:01:26,448 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:26,942 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:01:26,943 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:01:26,944 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:01:26,945 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:01:27,609 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:27,609 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:01:28,611 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:31,497 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:31,615 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:35,620 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:37,093 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:39,625 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:40,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:01:40,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:01:40,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:01:41,628 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:42,310 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:45,634 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:47,915 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:49,639 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:50,572 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:01:53,132 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:53,644 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:55,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:01:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:01:55,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:01:57,650 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:01:58,727 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:01:59,652 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:02,945 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:02:02,947 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:02:02,947 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:02:02,949 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:02:03,659 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:03,659 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:02:03,950 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:04,660 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:07,664 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:09,549 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:10,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:02:10,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:02:10,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:02:11,669 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:14,750 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:15,675 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:17,677 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:20,345 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:20,574 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:02:21,683 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:25,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:02:25,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:02:25,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:02:25,418 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:25,688 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:29,693 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:31,181 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:33,698 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:35,701 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:36,400 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:39,023 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:02:39,024 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:02:39,024 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:02:39,026 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:02:39,707 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:39,707 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:02:40,348 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:02:40,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:02:40,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:02:40,708 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:41,400 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:43,712 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:47,226 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:47,718 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:50,576 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:02:51,723 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:52,577 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:02:53,726 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:02:55,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:02:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:02:57,731 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:02:58,047 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:01,736 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:03,651 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:05,742 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:08,868 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:09,748 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:10,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:03:10,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:03:10,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:03:11,750 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:14,469 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:15,092 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:03:15,093 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:03:15,094 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:03:15,096 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:03:15,757 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:15,757 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:03:16,758 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:19,706 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:19,762 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:20,577 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:03:23,767 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:25,309 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:25,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:03:25,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:03:25,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:03:27,773 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:30,545 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:31,778 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:33,781 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:36,152 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:37,786 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:40,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:03:40,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:03:40,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:03:41,404 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:41,791 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:45,797 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:46,979 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:49,802 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:50,579 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:03:51,183 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:03:51,184 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:03:51,185 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:03:51,187 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:03:51,806 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:51,806 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:03:52,187 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:52,807 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:03:55,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:03:55,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:03:55,811 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:03:57,472 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:03:59,817 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:03,006 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:03,822 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:07,828 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:08,620 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:09,830 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:10,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:04:10,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:04:10,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:04:13,836 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:13,836 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:17,841 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:19,433 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:20,581 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:04:21,847 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:24,633 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:25,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:04:25,349 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:04:25,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:04:25,852 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:27,274 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:04:27,275 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:04:27,275 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:04:27,277 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:04:27,855 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:27,856 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:04:28,857 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:30,278 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:31,861 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:35,451 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:35,866 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:39,871 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:40,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:04:40,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:04:40,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:04:40,479 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:43,877 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:45,879 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:46,271 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:49,884 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:50,583 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:04:51,584 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:53,890 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:04:55,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:04:55,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:04:55,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:04:57,096 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:04:57,895 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:01,900 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:02,711 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:03,319 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:05:03,320 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:05:03,320 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:05:03,322 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:05:03,904 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:03,904 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:05:04,905 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:07,909 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:07,930 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:10,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:05:10,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:05:10,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:05:11,915 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:13,541 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:15,920 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:18,751 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:19,925 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:20,595 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:05:21,928 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:24,364 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:25,349 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:05:25,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:05:25,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:05:25,933 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:29,563 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:29,937 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:33,942 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:35,165 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:37,947 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:39,380 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:05:39,381 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:05:39,381 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:05:39,383 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:05:39,951 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:39,951 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:05:40,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:05:40,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:05:40,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:05:40,443 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:40,952 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:43,956 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:45,987 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:47,961 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:50,587 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:05:51,587 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:51,966 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:55,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:05:55,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:05:55,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:05:55,970 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:05:57,430 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:05:57,973 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:01,978 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:03,038 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:05,983 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:08,254 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:09,988 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:10,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:06:10,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:06:10,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:06:13,860 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:13,993 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:15,460 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:06:15,461 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:06:15,462 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:06:15,464 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:06:15,996 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:15,997 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:06:16,998 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:19,075 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:20,001 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:20,589 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:06:24,006 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:24,688 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:25,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:06:25,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:06:25,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:06:28,012 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:29,911 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:32,017 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:34,019 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:35,529 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:38,025 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:40,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:06:40,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:06:40,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:06:40,741 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:42,029 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:46,035 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:46,345 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:50,039 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:50,590 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:06:51,549 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:06:51,550 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:06:51,551 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:06:51,552 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:06:51,554 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:06:52,043 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:52,043 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:06:53,044 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:55,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:06:55,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:06:55,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:06:56,048 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:06:57,499 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:00,053 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:03,326 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:04,058 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:08,063 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:08,919 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:10,065 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:10,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:07:10,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:07:10,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:07:14,071 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:14,115 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:18,076 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:19,718 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:20,602 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:07:22,081 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:24,919 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:25,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:07:25,350 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:07:25,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:07:26,086 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:27,566 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:07:27,567 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:07:27,567 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:07:27,569 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:07:28,089 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:28,090 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:07:29,091 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:30,570 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:32,095 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:35,723 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:36,101 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:40,107 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:40,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:07:40,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:07:40,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:07:41,482 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:44,119 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:46,123 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:47,458 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:50,133 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:50,594 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:07:52,595 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:07:54,144 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:55,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:07:55,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:07:55,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:07:58,154 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:07:58,183 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:02,165 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:03,329 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:03,330 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:08:03,331 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:08:03,331 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:08:03,346 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:08:04,175 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:04,175 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:08:05,177 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:08,184 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:08,914 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:10,350 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:08:10,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:08:10,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:08:12,193 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:14,072 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:16,203 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:19,644 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:20,210 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:20,596 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:08:22,213 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:24,790 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:25,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:08:25,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:08:25,352 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:08:26,218 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:30,223 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:30,353 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:34,228 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:35,552 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:36,230 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:39,143 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:08:39,144 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:08:39,144 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:08:39,146 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:08:39,235 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:08:40,236 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:40,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:08:40,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:08:40,353 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:08:41,238 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:41,422 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:44,242 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:47,289 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:48,247 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:50,600 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:08:52,260 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:52,602 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:54,264 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:08:55,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:08:55,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:08:55,355 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:08:58,269 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:08:58,272 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:02,277 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:03,969 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:06,282 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:09,260 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:10,287 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:10,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:09:10,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:09:10,355 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:09:14,292 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:14,879 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:15,517 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:09:15,520 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:09:15,520 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:09:15,524 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:09:16,295 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:16,296 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:09:17,297 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:20,151 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:20,301 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:20,601 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:09:24,306 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:25,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:09:25,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:09:25,353 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:09:25,438 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:28,311 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:31,008 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:32,315 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:36,320 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:36,632 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:38,323 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:40,351 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:09:40,351 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:09:40,353 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:09:41,854 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:42,328 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:46,333 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:47,458 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:50,338 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:50,602 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:09:51,687 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:09:51,689 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:09:51,689 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:09:51,691 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:09:52,341 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:09:52,692 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:09:53,343 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:54,344 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:55,420 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:09:55,421 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:09:55,421 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:09:56,347 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:09:58,569 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:00,352 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:04,357 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:04,511 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:08,362 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:09,721 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:10,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:10:10,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:10:10,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:10:12,367 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:14,369 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:15,328 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:18,374 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:20,510 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:20,604 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:10:22,380 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:25,383 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:10:25,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:10:25,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:10:26,387 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:26,498 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:27,754 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:10:27,755 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:10:27,755 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:10:27,757 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:10:28,390 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:10:29,391 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:30,393 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:32,302 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:32,395 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:36,400 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:37,916 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:40,383 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:10:40,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:10:40,405 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:40,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:10:43,116 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:44,411 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:48,416 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:48,718 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:50,418 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:50,606 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:10:53,902 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:10:54,423 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:55,383 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:10:55,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:10:55,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:10:58,428 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:10:59,501 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:02,433 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:03,689 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:11:03,691 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:11:03,691 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:11:03,693 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:11:04,437 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:11:04,693 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:05,438 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:06,440 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:08,442 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:10,295 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:10,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:11:10,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:11:10,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:11:12,447 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:15,474 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:16,452 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:20,457 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:20,608 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:11:20,609 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:24,462 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:25,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:11:25,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:11:25,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:11:26,465 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:26,470 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:30,470 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:31,839 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:34,475 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:37,033 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:38,480 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:39,627 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:11:39,628 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:11:39,628 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:11:39,630 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:11:40,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:11:40,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:11:40,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:11:40,483 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:40,483 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:11:42,527 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:44,488 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:47,808 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:48,493 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:50,619 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:11:52,498 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:53,620 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:11:55,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:11:55,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:11:55,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:11:56,503 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:58,505 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:11:59,589 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:02,510 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:04,771 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:06,515 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:10,369 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:10,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:12:10,384 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:12:10,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:12:10,520 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:14,525 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:15,549 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:15,550 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:12:15,551 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:12:15,552 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:12:15,554 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:12:16,530 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:16,530 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:12:20,534 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:20,612 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:12:20,613 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:24,539 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:25,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:12:25,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:12:25,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:12:26,333 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:28,544 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:31,919 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:32,549 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:34,552 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:37,107 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:38,557 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:40,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:12:40,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:12:40,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:12:42,562 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:42,706 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:46,567 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:47,896 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:50,571 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:50,614 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:12:51,492 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:12:51,493 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:12:51,494 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:12:51,496 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:12:51,574 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:12:52,575 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:53,496 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:12:55,384 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:12:55,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:12:55,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:12:56,580 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:12:58,690 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:00,585 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:04,277 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:04,590 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:08,595 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:09,448 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:10,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:13:10,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:13:10,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:13:10,598 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:14,603 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:15,033 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:18,608 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:20,216 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:20,616 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:13:22,613 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:25,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:13:25,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:13:25,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:13:25,501 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:26,618 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:27,430 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:13:27,431 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:13:27,432 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:13:27,433 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:13:27,620 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:13:28,621 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:30,976 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:32,626 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:36,561 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:36,631 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:40,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:13:40,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:13:40,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:13:40,636 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:41,750 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:44,641 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:46,644 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:47,330 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:50,628 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:13:50,649 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:52,515 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:54,654 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:13:55,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:13:55,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:13:55,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:13:58,110 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:13:58,659 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:02,664 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:03,307 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:03,308 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:14:03,309 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:14:03,309 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:14:03,311 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:14:03,666 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:14:04,667 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:08,672 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:08,892 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:10,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:14:10,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:14:10,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:14:12,678 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:14,071 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:16,683 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:18,685 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:19,651 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:20,619 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:14:22,690 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:24,814 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:25,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:14:25,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:14:25,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:14:26,695 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:30,407 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:30,700 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:34,705 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:35,707 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:38,711 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:39,365 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:14:39,366 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:14:39,366 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:14:39,368 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:14:39,713 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:14:40,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:14:40,385 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:14:40,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:14:40,715 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:41,513 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:44,720 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:46,636 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:48,724 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:50,621 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:14:51,729 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:52,622 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:14:55,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:14:55,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:14:55,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:14:55,734 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:57,736 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:14:58,544 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:01,741 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:04,165 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:05,746 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:09,426 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:09,751 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:10,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:15:10,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:15:10,408 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:15:13,756 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:15,048 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:15,680 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:15:15,681 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:15:15,681 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:15:15,683 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:15:15,760 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:15,760 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:15:16,761 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:19,765 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:20,298 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:20,623 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:15:23,770 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:25,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:15:25,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:15:25,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:15:25,508 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:27,775 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:29,778 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:30,539 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:31,781 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:33,783 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:35,570 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:35,786 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:37,789 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:39,791 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:40,385 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:15:40,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:15:40,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:15:40,590 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:41,794 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:43,797 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:45,601 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:45,799 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:47,802 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:49,804 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:50,626 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:15:50,627 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:51,807 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:53,810 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:55,386 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:15:55,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:15:55,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:15:55,640 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:15:55,812 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:57,215 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:15:57,219 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 18:15:57,220 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 18:15:57,220 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 18:15:57,221 DEBUG SenderThread:239784 [sender.py:send():382] send: metric 2024-02-01 18:15:57,221 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:15:57,221 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:15:57,223 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:15:57,816 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:57,816 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:15:58,817 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:15:59,819 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:01,327 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:01,821 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:02,823 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:06,828 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:06,938 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:09,832 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:10,386 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:16:10,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:16:10,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:16:12,143 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:12,837 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/config.yaml 2024-02-01 18:16:13,838 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:15,841 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:17,724 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:19,846 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:20,627 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:16:22,891 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:23,851 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:25,386 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:16:25,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:16:25,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:16:27,856 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:28,490 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:31,861 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:33,645 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:33,646 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:16:33,647 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:16:33,647 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:16:33,650 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:16:33,865 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:33,865 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:16:34,866 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:37,870 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:39,229 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:40,386 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:16:40,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:16:40,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:16:41,875 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:44,390 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:45,881 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:49,886 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:49,982 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:50,629 DEBUG SenderThread:239784 [sender.py:send():382] send: stats 2024-02-01 18:16:51,888 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:55,146 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:16:55,386 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:16:55,386 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:16:55,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:16:55,893 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:16:59,899 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:17:00,735 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:17:03,904 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:17:05,905 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:17:07,909 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:17:09,494 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: partial_history 2024-02-01 18:17:09,495 DEBUG SenderThread:239784 [sender.py:send():382] send: history 2024-02-01 18:17:09,495 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: summary_record 2024-02-01 18:17:09,497 INFO SenderThread:239784 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end 2024-02-01 18:17:09,913 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:17:09,913 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/wandb-summary.json 2024-02-01 18:17:10,386 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: stop_status 2024-02-01 18:17:10,387 DEBUG SenderThread:239784 [sender.py:send_request():409] send_request: stop_status 2024-02-01 18:17:10,409 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-01 18:17:10,914 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log 2024-02-01 18:17:11,515 DEBUG HandlerThread:239784 [handler.py:handle_request():146] handle_request: status_report 2024-02-01 18:17:13,918 INFO Thread-12 :239784 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_175850-i93q0p12/files/output.log