2024-03-22 04:34:07,566 INFO    StreamThr :4275 [internal.py:wandb_internal():86] W&B internal server running at pid: 4275, started at: 2024-03-22 04:34:07.562793
2024-03-22 04:34:07,589 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status
2024-03-22 04:34:12,594 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:12,594 INFO    WriterThread:4275 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/run-hize5f4p.wandb
2024-03-22 04:34:12,609 DEBUG   SenderThread:4275 [sender.py:send():378] send: header
2024-03-22 04:34:12,616 DEBUG   SenderThread:4275 [sender.py:send():378] send: run
2024-03-22 04:34:13,047 INFO    SenderThread:4275 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files
2024-03-22 04:34:13,047 INFO    SenderThread:4275 [sender.py:_start_run_threads():1099] run started: hize5f4p with start time 1711082047.565462
2024-03-22 04:34:13,064 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: check_version
2024-03-22 04:34:13,064 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: check_version
2024-03-22 04:34:13,182 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: run_start
2024-03-22 04:34:13,192 DEBUG   HandlerThread:4275 [system_info.py:__init__():26] System info init
2024-03-22 04:34:13,193 DEBUG   HandlerThread:4275 [system_info.py:__init__():41] System info init done
2024-03-22 04:34:13,193 INFO    HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor
2024-03-22 04:34:13,193 INFO    SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-22 04:34:13,193 INFO    HandlerThread:4275 [system_monitor.py:probe():214] Collecting system info
2024-03-22 04:34:13,194 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring
2024-03-22 04:34:13,195 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring
2024-03-22 04:34:13,196 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring
2024-03-22 04:34:13,197 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring
2024-03-22 04:34:13,198 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring
2024-03-22 04:34:13,203 DEBUG   HandlerThread:4275 [system_info.py:probe():150] Probing system
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [system_info.py:probe():198] Probing system done
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-22T04:34:13.203984', 'startedAt': '2024-03-22T04:34:07.465777', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1yggNDPxtSX3bri4ON4ppYc-G-Q2zFtkQ', 'host': '6b98cdd15d10', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.876033782958984}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}}
2024-03-22 04:34:13,208 INFO    HandlerThread:4275 [system_monitor.py:probe():224] Finished collecting system info
2024-03-22 04:34:13,208 INFO    HandlerThread:4275 [system_monitor.py:probe():227] Publishing system info
2024-03-22 04:34:13,226 INFO    HandlerThread:4275 [system_monitor.py:probe():229] Finished publishing system info
2024-03-22 04:34:13,232 DEBUG   SenderThread:4275 [sender.py:send():378] send: files
2024-03-22 04:34:13,233 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now
2024-03-22 04:34:13,733 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: python_packages
2024-03-22 04:34:13,733 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: python_packages
2024-03-22 04:34:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:13,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:13,741 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:13,888 INFO    wandb-upload_0:4275 [upload_job.py:push():131] Uploaded file /tmp/tmp3i0vj0_swandb/55hqq2eg-wandb-metadata.json
2024-03-22 04:34:13,972 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:13,973 DEBUG   SenderThread:4275 [sender.py:send():378] send: config
2024-03-22 04:34:13,975 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:34:13,975 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:13,976 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:34:13,976 WARNING SenderThread:4275 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-03-22 04:34:13,976 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:14,052 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/requirements.txt
2024-03-22 04:34:14,053 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-metadata.json
2024-03-22 04:34:17,977 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:22,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:27,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:33,912 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:38,918 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:40,099 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml
2024-03-22 04:34:43,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:43,736 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:43,776 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:44,947 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:49,948 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:54,949 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:58,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:58,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:58,774 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:00,929 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:05,930 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:10,931 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:13,199 DEBUG   SystemMonitor:4275 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-22 04:35:13,200 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:35:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:13,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:13,774 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:16,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:21,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:26,923 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:28,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:31,951 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:36,952 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:41,952 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:43,201 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:35:43,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:43,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:43,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:47,914 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:52,914 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:57,915 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:58,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:58,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:58,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:02,958 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:07,958 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:12,959 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:13,202 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:36:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:13,736 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:13,776 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:18,925 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:23,926 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:28,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:28,777 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:28,959 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:33,960 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:38,377 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:36:38,384 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,386 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:36:38,386 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:36:38,395 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:36:39,314 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:36:39,396 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:43,202 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:36:49,784 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:50,358 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml
2024-03-22 04:36:50,399 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:50,401 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:50,402 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:55,622 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,752 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,753 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,753 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:14,753 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:37:14,754 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:14,783 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:37:16,284 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:21,289 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,399 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,407 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:31,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:37:31,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,409 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:36,568 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:41,569 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:43,204 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:37:44,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:44,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:46,973 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:51,974 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:56,975 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report