| 2024-03-22 04:34:07,566 INFO StreamThr :4275 [internal.py:wandb_internal():86] W&B internal server running at pid: 4275, started at: 2024-03-22 04:34:07.562793 | |
| 2024-03-22 04:34:07,589 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-22 04:34:12,594 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:12,594 INFO WriterThread:4275 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/run-hize5f4p.wandb | |
| 2024-03-22 04:34:12,609 DEBUG SenderThread:4275 [sender.py:send():378] send: header | |
| 2024-03-22 04:34:12,616 DEBUG SenderThread:4275 [sender.py:send():378] send: run | |
| 2024-03-22 04:34:13,047 INFO SenderThread:4275 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files | |
| 2024-03-22 04:34:13,047 INFO SenderThread:4275 [sender.py:_start_run_threads():1099] run started: hize5f4p with start time 1711082047.565462 | |
| 2024-03-22 04:34:13,064 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-22 04:34:13,064 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: check_version | |
| 2024-03-22 04:34:13,182 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-22 04:34:13,192 DEBUG HandlerThread:4275 [system_info.py:__init__():26] System info init | |
| 2024-03-22 04:34:13,193 DEBUG HandlerThread:4275 [system_info.py:__init__():41] System info init done | |
| 2024-03-22 04:34:13,193 INFO HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-22 04:34:13,193 INFO SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-22 04:34:13,193 INFO HandlerThread:4275 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-22 04:34:13,194 INFO SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-22 04:34:13,195 INFO SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-22 04:34:13,196 INFO SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-22 04:34:13,197 INFO SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-22 04:34:13,198 INFO SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-22 04:34:13,203 DEBUG HandlerThread:4275 [system_info.py:probe():150] Probing system | |
| 2024-03-22 04:34:13,208 DEBUG HandlerThread:4275 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-22 04:34:13,208 DEBUG HandlerThread:4275 [system_info.py:probe():198] Probing system done | |
| 2024-03-22 04:34:13,208 DEBUG HandlerThread:4275 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-22T04:34:13.203984', 'startedAt': '2024-03-22T04:34:07.465777', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1yggNDPxtSX3bri4ON4ppYc-G-Q2zFtkQ', 'host': '6b98cdd15d10', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.876033782958984}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}} | |
| 2024-03-22 04:34:13,208 INFO HandlerThread:4275 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-22 04:34:13,208 INFO HandlerThread:4275 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-22 04:34:13,226 INFO HandlerThread:4275 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-22 04:34:13,232 DEBUG SenderThread:4275 [sender.py:send():378] send: files | |
| 2024-03-22 04:34:13,233 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-03-22 04:34:13,733 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-22 04:34:13,733 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: python_packages | |
| 2024-03-22 04:34:13,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:13,735 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:34:13,741 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:13,888 INFO wandb-upload_0:4275 [upload_job.py:push():131] Uploaded file /tmp/tmp3i0vj0_swandb/55hqq2eg-wandb-metadata.json | |
| 2024-03-22 04:34:13,972 DEBUG SenderThread:4275 [sender.py:send():378] send: telemetry | |
| 2024-03-22 04:34:13,973 DEBUG SenderThread:4275 [sender.py:send():378] send: config | |
| 2024-03-22 04:34:13,975 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:34:13,975 DEBUG SenderThread:4275 [sender.py:send():378] send: telemetry | |
| 2024-03-22 04:34:13,976 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:34:13,976 WARNING SenderThread:4275 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-22 04:34:13,976 DEBUG SenderThread:4275 [sender.py:send():378] send: telemetry | |
| 2024-03-22 04:34:14,052 INFO Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/requirements.txt | |
| 2024-03-22 04:34:14,053 INFO Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-metadata.json | |
| 2024-03-22 04:34:17,977 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:22,978 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:27,978 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:28,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:28,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:34:28,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:33,912 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:38,918 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:40,099 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml | |
| 2024-03-22 04:34:43,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:43,736 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:43,776 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:34:44,947 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:49,948 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:54,949 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:58,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:58,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:58,774 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:00,929 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:05,930 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:10,931 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:13,199 DEBUG SystemMonitor:4275 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-22 04:35:13,200 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:35:13,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:13,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:13,774 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:16,922 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:21,922 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:26,923 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:28,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:28,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:28,775 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:31,951 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:36,952 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:41,952 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:43,201 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:35:43,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:43,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:43,775 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:47,914 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:52,914 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:57,915 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:58,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:58,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:58,775 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:02,958 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:07,958 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:12,959 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:13,202 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:36:13,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:36:13,736 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:36:13,776 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:18,925 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:23,926 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:28,735 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:36:28,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:36:28,777 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:28,959 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:33,960 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:38,377 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:36:38,384 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,385 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,385 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,385 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,386 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:36:38,386 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:36:38,395 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:36:39,314 INFO Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:36:39,396 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:43,202 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:36:49,784 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:50,358 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml | |
| 2024-03-22 04:36:50,399 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:50,401 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:36:50,402 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:36:55,622 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,735 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,752 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,753 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,753 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:14,753 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:37:14,754 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:37:14,783 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:37:16,284 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:21,289 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:31,399 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:31,407 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:31,408 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:37:31,408 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:31,409 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:37:36,568 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:41,569 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:43,204 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:37:44,743 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:44,744 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:37:46,973 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:51,974 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:56,975 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:59,743 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:59,743 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:38:01,989 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:06,990 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:11,990 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:13,205 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:38:14,743 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:38:14,744 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:38:17,990 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:22,991 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:27,992 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:29,743 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:38:29,744 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:38:33,963 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:38,964 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:43,206 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:38:44,207 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:44,744 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:38:44,744 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:38:49,987 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:54,988 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:38:59,743 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:38:59,744 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:39:00,966 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:05,966 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:10,497 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:39:10,500 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:39:10,501 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:39:10,506 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:39:10,715 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:39:11,508 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:13,206 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:39:17,208 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:20,656 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:39:20,657 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:39:22,874 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:27,875 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:37,621 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:37,629 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:39:37,630 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:39:38,874 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:43,207 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:39:44,209 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:49,213 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:52,629 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:39:52,629 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:39:54,879 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:39:59,880 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:04,881 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:07,628 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:40:07,629 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:40:10,794 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:13,208 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:40:16,209 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:21,210 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:22,628 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:40:22,629 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:40:26,863 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:31,864 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:36,865 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:37,629 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:40:37,630 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:40:42,807 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:43,209 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:40:48,210 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:52,629 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:40:52,630 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:40:53,792 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:40:55,007 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:40:55,008 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:40:55,008 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:40:55,023 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:40:56,020 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:40:59,026 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:07,713 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:08,198 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:41:08,199 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:41:09,376 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:13,209 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:41:29,703 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:29,719 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:41:29,719 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:41:34,921 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:39,922 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:43,210 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:41:44,706 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:41:44,722 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:41:44,924 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:49,925 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:54,926 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:41:59,706 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:41:59,707 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:42:00,919 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:05,920 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:10,921 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:13,211 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:42:14,706 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:42:14,707 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:42:16,859 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:21,860 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:26,861 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:29,706 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:42:29,707 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:42:31,901 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:36,902 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:41,903 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:43,212 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:42:44,707 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:42:44,707 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:42:46,919 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:51,919 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:56,920 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:42:59,707 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:42:59,707 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:43:01,947 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:06,947 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:11,948 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:13,213 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:43:14,707 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:43:14,708 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:43:17,865 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:22,865 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:27,866 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:29,707 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:43:29,708 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:43:33,852 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:35,932 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:43:35,934 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:43:35,935 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:43:35,942 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:43:36,437 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:43:38,944 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:43,214 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:43:46,755 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:47,270 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:43:47,271 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:43:52,485 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:43:59,193 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:02,270 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:44:02,271 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:44:03,454 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:08,455 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:13,215 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:44:14,216 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:17,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:44:17,271 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:44:19,504 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:24,505 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:29,506 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:32,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:44:32,283 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:44:35,488 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:40,489 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:43,215 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:44:46,219 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:47,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:44:47,271 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:44:51,474 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:44:56,475 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:01,475 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:02,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:45:02,272 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:45:07,407 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:12,408 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:13,216 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:45:17,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:45:17,271 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:45:17,451 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:22,452 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:27,453 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:32,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:45:32,272 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:45:33,416 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:38,417 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:43,217 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:45:44,218 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:47,271 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:45:47,272 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:45:49,508 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:54,509 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:45:59,256 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:45:59,260 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:45:59,261 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:45:59,266 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:45:59,975 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:46:00,268 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:05,269 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:10,661 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:11,230 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:46:11,230 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:46:13,218 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:46:16,219 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:30,809 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:30,810 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:30,817 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:46:30,818 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:46:31,977 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:36,978 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:41,979 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:43,219 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:46:45,817 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:46:45,818 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:46:47,978 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:52,979 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:46:57,980 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:00,817 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:47:00,818 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:47:03,064 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:08,064 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:13,065 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:13,220 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:47:15,817 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:47:15,818 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:47:19,024 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:24,025 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:29,029 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:30,817 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:47:30,818 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:47:34,998 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:39,998 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:43,220 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:47:45,222 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:45,817 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:47:45,818 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:47:50,999 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:47:54,054 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:47:54,057 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:47:54,058 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:47:54,064 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:47:54,251 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:47:56,066 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:04,801 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:05,680 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:48:05,681 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:48:06,885 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:11,889 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:16,895 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:16,896 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:48:20,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:48:20,683 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:48:20,840 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:48:20,928 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:48:20,934 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:48:20,935 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:48:20,941 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:48:20,942 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:48:20,951 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:48:20,952 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:48:20,960 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:48:20,962 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:48:20,969 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:48:20,971 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:48:20,971 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:48:21,160 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:48:21,480 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:48:22,163 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:27,164 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:32,165 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:34,363 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-22 04:48:34,363 INFO HandlerThread:4275 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-22 04:48:34,363 INFO HandlerThread:4275 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-22 04:48:34,363 DEBUG SystemMonitor:4275 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-22 04:48:34,367 DEBUG SystemMonitor:4275 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-22 04:48:34,367 INFO HandlerThread:4275 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-22 04:48:34,376 INFO HandlerThread:4275 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-22 04:48:34,381 INFO HandlerThread:4275 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-22 04:48:34,383 INFO HandlerThread:4275 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-22 04:48:34,383 INFO HandlerThread:4275 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-22 04:48:34,384 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:48:35,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:48:35,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:48:37,861 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:42,861 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:47,862 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:50,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:48:50,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:48:52,893 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:48:57,894 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:02,895 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:05,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:49:05,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:49:08,854 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:13,855 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:18,856 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:20,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:49:20,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:49:23,927 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:28,928 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:33,929 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:35,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:49:35,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:49:39,888 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:44,889 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:49,890 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:49:50,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:49:50,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:49:55,837 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:00,838 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:05,681 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:50:05,682 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:50:05,876 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:10,878 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:15,879 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:20,682 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:50:20,683 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:50:21,843 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:26,844 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:31,845 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:35,682 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:50:35,683 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:50:36,879 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:41,880 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:46,881 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:50,682 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:50:50,683 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:50:51,886 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:50:56,887 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:51:01,888 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:51:05,682 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:51:05,683 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:51:06,907 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:51:09,923 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-22 04:51:09,924 INFO HandlerThread:4275 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-22 04:51:09,925 INFO HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-22 04:51:09,925 INFO SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-22 04:51:09,926 INFO SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-22 04:51:09,927 INFO SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-22 04:51:09,929 INFO SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-22 04:51:09,930 INFO SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-22 04:51:09,935 INFO SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-22 04:51:11,908 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |