2024-03-22 04:34:07,566 INFO    StreamThr :4275 [internal.py:wandb_internal():86] W&B internal server running at pid: 4275, started at: 2024-03-22 04:34:07.562793
2024-03-22 04:34:07,589 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status
2024-03-22 04:34:12,594 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:12,594 INFO    WriterThread:4275 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/run-hize5f4p.wandb
2024-03-22 04:34:12,609 DEBUG   SenderThread:4275 [sender.py:send():378] send: header
2024-03-22 04:34:12,616 DEBUG   SenderThread:4275 [sender.py:send():378] send: run
2024-03-22 04:34:13,047 INFO    SenderThread:4275 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files
2024-03-22 04:34:13,047 INFO    SenderThread:4275 [sender.py:_start_run_threads():1099] run started: hize5f4p with start time 1711082047.565462
2024-03-22 04:34:13,064 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: check_version
2024-03-22 04:34:13,064 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: check_version
2024-03-22 04:34:13,182 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: run_start
2024-03-22 04:34:13,192 DEBUG   HandlerThread:4275 [system_info.py:__init__():26] System info init
2024-03-22 04:34:13,193 DEBUG   HandlerThread:4275 [system_info.py:__init__():41] System info init done
2024-03-22 04:34:13,193 INFO    HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor
2024-03-22 04:34:13,193 INFO    SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-22 04:34:13,193 INFO    HandlerThread:4275 [system_monitor.py:probe():214] Collecting system info
2024-03-22 04:34:13,194 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring
2024-03-22 04:34:13,195 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring
2024-03-22 04:34:13,196 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring
2024-03-22 04:34:13,197 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring
2024-03-22 04:34:13,198 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring
2024-03-22 04:34:13,203 DEBUG   HandlerThread:4275 [system_info.py:probe():150] Probing system
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [system_info.py:probe():198] Probing system done
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-22T04:34:13.203984', 'startedAt': '2024-03-22T04:34:07.465777', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1yggNDPxtSX3bri4ON4ppYc-G-Q2zFtkQ', 'host': '6b98cdd15d10', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.876033782958984}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}}
2024-03-22 04:34:13,208 INFO    HandlerThread:4275 [system_monitor.py:probe():224] Finished collecting system info
2024-03-22 04:34:13,208 INFO    HandlerThread:4275 [system_monitor.py:probe():227] Publishing system info
2024-03-22 04:34:13,226 INFO    HandlerThread:4275 [system_monitor.py:probe():229] Finished publishing system info
2024-03-22 04:34:13,232 DEBUG   SenderThread:4275 [sender.py:send():378] send: files
2024-03-22 04:34:13,233 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now
2024-03-22 04:34:13,733 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: python_packages
2024-03-22 04:34:13,733 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: python_packages
2024-03-22 04:34:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:13,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:13,741 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:13,888 INFO    wandb-upload_0:4275 [upload_job.py:push():131] Uploaded file /tmp/tmp3i0vj0_swandb/55hqq2eg-wandb-metadata.json
2024-03-22 04:34:13,972 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:13,973 DEBUG   SenderThread:4275 [sender.py:send():378] send: config
2024-03-22 04:34:13,975 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:34:13,975 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:13,976 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:34:13,976 WARNING SenderThread:4275 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-03-22 04:34:13,976 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:14,052 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/requirements.txt
2024-03-22 04:34:14,053 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-metadata.json
2024-03-22 04:34:17,977 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:22,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:27,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:33,912 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:38,918 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:40,099 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml
2024-03-22 04:34:43,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:43,736 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:43,776 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:44,947 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:49,948 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:54,949 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:58,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:58,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:58,774 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:00,929 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:05,930 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:10,931 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:13,199 DEBUG   SystemMonitor:4275 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-22 04:35:13,200 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:35:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:13,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:13,774 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:16,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:21,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:26,923 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:28,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:31,951 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:36,952 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:41,952 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:43,201 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:35:43,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:43,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:43,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:47,914 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:52,914 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:57,915 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:58,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:58,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:58,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:02,958 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:07,958 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:12,959 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:13,202 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:36:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:13,736 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:13,776 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:18,925 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:23,926 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:28,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:28,777 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:28,959 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:33,960 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:38,377 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:36:38,384 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,386 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:36:38,386 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:36:38,395 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:36:39,314 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:36:39,396 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:43,202 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:36:49,784 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:50,358 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml
2024-03-22 04:36:50,399 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:50,401 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:50,402 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:55,622 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,752 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,753 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,753 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:14,753 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:37:14,754 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:14,783 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:37:16,284 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:21,289 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,399 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,407 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:31,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:37:31,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,409 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:36,568 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:41,569 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:43,204 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:37:44,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:44,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:46,973 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:51,974 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:56,975 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:59,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:59,743 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:01,989 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:06,990 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:11,990 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:13,205 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:38:14,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:14,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:17,990 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:22,991 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:27,992 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:29,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:29,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:33,963 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:38,964 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:43,206 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:38:44,207 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:44,744 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:44,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:49,987 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:54,988 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:59,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:59,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:00,966 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:05,966 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:10,497 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:39:10,500 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:39:10,501 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:39:10,506 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:39:10,715 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:39:11,508 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:13,206 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:39:17,208 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:20,656 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:39:20,657 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:22,874 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:27,875 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:37,621 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:37,629 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:39:37,630 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:38,874 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:43,207 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:39:44,209 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:49,213 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:52,629 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:39:52,629 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:54,879 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:59,880 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:04,881 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:07,628 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:40:07,629 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:40:10,794 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:13,208 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:40:16,209 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:21,210 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:22,628 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:40:22,629 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:40:26,863 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:31,864 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:36,865 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:37,629 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:40:37,630 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:40:42,807 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:43,209 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:40:48,210 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:52,629 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:40:52,630 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:40:53,792 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:40:55,007 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:40:55,008 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:40:55,008 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:40:55,023 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:40:56,020 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:40:59,026 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:07,713 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:08,198 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:41:08,199 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:41:09,376 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:13,209 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:41:29,703 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:29,719 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:41:29,719 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:41:34,921 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:39,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:43,210 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:41:44,706 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:41:44,722 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:41:44,924 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:49,925 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:54,926 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:41:59,706 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:41:59,707 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:42:00,919 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:05,920 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:10,921 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:13,211 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:42:14,706 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:42:14,707 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:42:16,859 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:21,860 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:26,861 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:29,706 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:42:29,707 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:42:31,901 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:36,902 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:41,903 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:43,212 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:42:44,707 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:42:44,707 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:42:46,919 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:51,919 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:56,920 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:42:59,707 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:42:59,707 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:43:01,947 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:06,947 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:11,948 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:13,213 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:43:14,707 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:43:14,708 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:43:17,865 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:22,865 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:27,866 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:29,707 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:43:29,708 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:43:33,852 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:35,932 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:43:35,934 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:43:35,935 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:43:35,942 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:43:36,437 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:43:38,944 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:43,214 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:43:46,755 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:47,270 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:43:47,271 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:43:52,485 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:43:59,193 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:02,270 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:44:02,271 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:44:03,454 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:08,455 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:13,215 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:44:14,216 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:17,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:44:17,271 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:44:19,504 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:24,505 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:29,506 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:32,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:44:32,283 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:44:35,488 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:40,489 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:43,215 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:44:46,219 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:47,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:44:47,271 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:44:51,474 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:44:56,475 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:01,475 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:02,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:45:02,272 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:45:07,407 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:12,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:13,216 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:45:17,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:45:17,271 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:45:17,451 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:22,452 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:27,453 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:32,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:45:32,272 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:45:33,416 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:38,417 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:43,217 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:45:44,218 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:47,271 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:45:47,272 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:45:49,508 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:54,509 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:45:59,256 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:45:59,260 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:45:59,261 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:45:59,266 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:45:59,975 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:46:00,268 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:05,269 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:10,661 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:11,230 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:46:11,230 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:46:13,218 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:46:16,219 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:30,809 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:30,810 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:30,817 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:46:30,818 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:46:31,977 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:36,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:41,979 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:43,219 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:46:45,817 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:46:45,818 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:46:47,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:52,979 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:46:57,980 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:00,817 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:47:00,818 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:47:03,064 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:08,064 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:13,065 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:13,220 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:47:15,817 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:47:15,818 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:47:19,024 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:24,025 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:29,029 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:30,817 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:47:30,818 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:47:34,998 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:39,998 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:43,220 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:47:45,222 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:45,817 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:47:45,818 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:47:50,999 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:47:54,054 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:47:54,057 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:47:54,058 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:47:54,064 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:47:54,251 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:47:56,066 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:04,801 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:05,680 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:48:05,681 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:48:06,885 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:11,889 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:16,895 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:16,896 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:48:20,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:48:20,683 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:48:20,840 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:48:20,928 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:48:20,934 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:48:20,935 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:48:20,941 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:48:20,942 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:48:20,951 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:48:20,952 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:48:20,960 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:48:20,962 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:48:20,969 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:48:20,971 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:48:20,971 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:48:21,160 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:48:21,480 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:48:22,163 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:27,164 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:32,165 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:34,363 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: pause
2024-03-22 04:48:34,363 INFO    HandlerThread:4275 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-03-22 04:48:34,363 INFO    HandlerThread:4275 [system_monitor.py:finish():203] Stopping system monitor
2024-03-22 04:48:34,363 DEBUG   SystemMonitor:4275 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-03-22 04:48:34,367 DEBUG   SystemMonitor:4275 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-03-22 04:48:34,367 INFO    HandlerThread:4275 [interfaces.py:finish():202] Joined cpu monitor
2024-03-22 04:48:34,376 INFO    HandlerThread:4275 [interfaces.py:finish():202] Joined disk monitor
2024-03-22 04:48:34,381 INFO    HandlerThread:4275 [interfaces.py:finish():202] Joined gpu monitor
2024-03-22 04:48:34,383 INFO    HandlerThread:4275 [interfaces.py:finish():202] Joined memory monitor
2024-03-22 04:48:34,383 INFO    HandlerThread:4275 [interfaces.py:finish():202] Joined network monitor
2024-03-22 04:48:34,384 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:48:35,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:48:35,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:48:37,861 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:42,861 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:47,862 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:50,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:48:50,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:48:52,893 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:48:57,894 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:02,895 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:05,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:49:05,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:49:08,854 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:13,855 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:18,856 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:20,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:49:20,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:49:23,927 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:28,928 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:33,929 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:35,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:49:35,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:49:39,888 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:44,889 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:49,890 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:49:50,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:49:50,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:49:55,837 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:00,838 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:05,681 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:50:05,682 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:50:05,876 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:10,878 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:15,879 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:20,682 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:50:20,683 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:50:21,843 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:26,844 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:31,845 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:35,682 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:50:35,683 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:50:36,879 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:41,880 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:46,881 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:50,682 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:50:50,683 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:50:51,886 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:50:56,887 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:51:01,888 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:51:05,682 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:51:05,683 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:51:06,907 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:51:09,923 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: resume
2024-03-22 04:51:09,924 INFO    HandlerThread:4275 [handler.py:handle_request_resume():699] starting system metrics thread
2024-03-22 04:51:09,925 INFO    HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor
2024-03-22 04:51:09,925 INFO    SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-22 04:51:09,926 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring
2024-03-22 04:51:09,927 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring
2024-03-22 04:51:09,929 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring
2024-03-22 04:51:09,930 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring
2024-03-22 04:51:09,935 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring
2024-03-22 04:51:11,908 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report