| 2024-03-21 01:54:58,388 INFO StreamThr :7197 [internal.py:wandb_internal():86] W&B internal server running at pid: 7197, started at: 2024-03-21 01:54:58.384167 | |
| 2024-03-21 01:54:58,394 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-21 01:55:03,409 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:03,409 INFO WriterThread:7197 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/run-y11xrjxv.wandb | |
| 2024-03-21 01:55:03,434 DEBUG SenderThread:7197 [sender.py:send():378] send: header | |
| 2024-03-21 01:55:03,442 DEBUG SenderThread:7197 [sender.py:send():378] send: run | |
| 2024-03-21 01:55:04,065 INFO SenderThread:7197 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files | |
| 2024-03-21 01:55:04,066 INFO SenderThread:7197 [sender.py:_start_run_threads():1099] run started: y11xrjxv with start time 1710986098.384977 | |
| 2024-03-21 01:55:04,082 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-21 01:55:04,083 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: check_version | |
| 2024-03-21 01:55:04,202 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-21 01:55:04,211 DEBUG HandlerThread:7197 [system_info.py:__init__():26] System info init | |
| 2024-03-21 01:55:04,211 DEBUG HandlerThread:7197 [system_info.py:__init__():41] System info init done | |
| 2024-03-21 01:55:04,211 INFO HandlerThread:7197 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-21 01:55:04,211 INFO SystemMonitor:7197 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-21 01:55:04,212 INFO HandlerThread:7197 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-21 01:55:04,212 INFO SystemMonitor:7197 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-21 01:55:04,213 INFO SystemMonitor:7197 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-21 01:55:04,214 INFO SystemMonitor:7197 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-21 01:55:04,215 INFO SystemMonitor:7197 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-21 01:55:04,215 INFO SystemMonitor:7197 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-21 01:55:04,228 DEBUG HandlerThread:7197 [system_info.py:probe():150] Probing system | |
| 2024-03-21 01:55:04,234 DEBUG HandlerThread:7197 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-21 01:55:04,234 DEBUG HandlerThread:7197 [system_info.py:probe():198] Probing system done | |
| 2024-03-21 01:55:04,234 DEBUG HandlerThread:7197 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-21T01:55:04.228731', 'startedAt': '2024-03-21T01:54:58.281206', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb%EC%9D%98%20%EC%82%AC%EB%B3%B8', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1glmddDRkqHDhfUT8slC17FnAssneTUkr', 'host': 'd72aea89d4b6', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.176, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.176, 'min': 0.0, 'max': 0.0}, {'current': 2000.176, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.871490478515625}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674781799316406}} | |
| 2024-03-21 01:55:04,235 INFO HandlerThread:7197 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-21 01:55:04,235 INFO HandlerThread:7197 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-21 01:55:04,254 INFO HandlerThread:7197 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-21 01:55:04,260 DEBUG SenderThread:7197 [sender.py:send():378] send: files | |
| 2024-03-21 01:55:04,260 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-03-21 01:55:04,647 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-21 01:55:04,648 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:55:04,648 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: python_packages | |
| 2024-03-21 01:55:04,649 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:55:04,656 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:55:05,019 DEBUG SenderThread:7197 [sender.py:send():378] send: telemetry | |
| 2024-03-21 01:55:05,020 DEBUG SenderThread:7197 [sender.py:send():378] send: config | |
| 2024-03-21 01:55:05,021 DEBUG SenderThread:7197 [sender.py:send():378] send: metric | |
| 2024-03-21 01:55:05,022 DEBUG SenderThread:7197 [sender.py:send():378] send: telemetry | |
| 2024-03-21 01:55:05,022 DEBUG SenderThread:7197 [sender.py:send():378] send: metric | |
| 2024-03-21 01:55:05,022 WARNING SenderThread:7197 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-21 01:55:05,022 DEBUG SenderThread:7197 [sender.py:send():378] send: telemetry | |
| 2024-03-21 01:55:05,070 INFO Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-metadata.json | |
| 2024-03-21 01:55:05,070 INFO Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/requirements.txt | |
| 2024-03-21 01:55:05,160 INFO wandb-upload_0:7197 [upload_job.py:push():131] Uploaded file /tmp/tmpe7x8kdblwandb/qivfhje6-wandb-metadata.json | |
| 2024-03-21 01:55:09,023 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:14,024 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:19,025 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:19,648 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:55:19,648 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:55:19,652 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:55:24,884 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:29,891 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:31,114 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/config.yaml | |
| 2024-03-21 01:55:34,648 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:55:34,648 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:55:34,689 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:55:35,977 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:40,977 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:45,980 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:49,648 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:55:49,649 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:55:49,689 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:55:50,983 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:55:55,984 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:00,985 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:04,216 DEBUG SystemMonitor:7197 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-21 01:56:04,217 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:56:04,648 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:56:04,649 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:56:04,689 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:56:06,724 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:11,725 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:16,726 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:19,648 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:56:19,649 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:56:19,689 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:56:21,983 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:26,984 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:31,985 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:34,218 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:56:34,650 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:56:34,651 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:56:34,691 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:56:37,948 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:42,949 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:47,949 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:49,650 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:56:49,651 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:56:49,691 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:56:52,960 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:56:57,961 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:02,961 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:04,219 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:57:04,650 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:57:04,651 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:57:04,691 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:57:07,993 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:12,994 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:17,995 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:19,650 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:57:19,651 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:57:19,691 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:57:23,885 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:28,885 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:33,886 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:34,219 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:57:34,650 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:57:34,651 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:57:34,697 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:57:36,218 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 01:57:36,221 DEBUG SenderThread:7197 [sender.py:send():378] send: metric | |
| 2024-03-21 01:57:36,221 DEBUG SenderThread:7197 [sender.py:send():378] send: metric | |
| 2024-03-21 01:57:36,221 DEBUG SenderThread:7197 [sender.py:send():378] send: metric | |
| 2024-03-21 01:57:36,222 DEBUG SenderThread:7197 [sender.py:send():378] send: metric | |
| 2024-03-21 01:57:36,224 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 01:57:36,225 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 01:57:36,230 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 01:57:36,349 INFO Thread-12 :7197 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 01:57:39,233 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:57:44,234 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:03,817 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:03,819 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:03,819 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:04,033 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:58:04,103 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:58:04,105 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:58:04,415 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:05,362 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:58:05,506 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/config.yaml | |
| 2024-03-21 01:58:10,366 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:15,367 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:20,047 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:58:20,049 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:58:20,049 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:58:21,305 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:26,306 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:31,307 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:53,904 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-21 01:58:53,911 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-21 01:58:53,911 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:53,911 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:58:53,913 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:53,913 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:58:53,913 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:53,913 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:58:53,913 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:58:53,904 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:58:53,914 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-21 01:58:55,052 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:58:58,517 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:00,166 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:03,518 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:04,222 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:59:05,167 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:09,223 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:10,169 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:14,224 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:15,170 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:19,225 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:20,171 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:24,225 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:25,172 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:29,226 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:30,173 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:34,222 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 01:59:35,176 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:35,223 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:40,177 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:40,224 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:45,178 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:45,225 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:50,179 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:50,226 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 01:59:55,179 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 01:59:55,227 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:00,180 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:00,228 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:04,223 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:00:05,181 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:06,224 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:10,182 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:11,226 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:15,183 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:16,227 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:20,184 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:21,228 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:25,185 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:26,229 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:30,187 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:31,230 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:34,224 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:00:35,188 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:00:37,010 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 02:00:37,013 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 02:00:37,014 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:00:37,015 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:00:37,019 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:00:37,893 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 02:00:42,021 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:05,507 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:05,510 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:05,510 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:05,510 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:05,511 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:01:05,717 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:07,512 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:11,560 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:12,513 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:17,514 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:20,668 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:22,516 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:26,471 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:27,517 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:31,472 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:32,518 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:34,225 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:01:36,473 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:38,227 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:41,476 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:43,228 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:46,479 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:48,229 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:51,480 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:53,229 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:01:56,482 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:01:58,230 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:01,483 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:03,232 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:04,226 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:02:06,484 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:09,230 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:11,490 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:14,231 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:16,491 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:19,232 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:21,492 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:24,232 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:26,494 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:29,234 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:31,495 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:34,227 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:02:35,228 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:36,496 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:39,633 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 02:02:39,634 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 02:02:39,635 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:02:39,645 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:02:40,359 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 02:02:40,651 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:41,496 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:02:45,651 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:50,652 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:58,007 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:02:58,222 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:00,654 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:04,228 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:03:11,188 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:11,195 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:16,191 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:16,981 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:21,191 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:22,003 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:26,192 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:27,009 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:31,193 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:32,010 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:34,229 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:03:36,230 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:37,011 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:41,233 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:42,012 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:46,234 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:47,012 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:51,235 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:52,013 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:03:56,236 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:03:57,014 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:01,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:02,015 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:04,230 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:04:07,017 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:07,236 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:12,018 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:12,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:17,020 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:17,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:22,021 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:22,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:27,022 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:27,240 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:32,022 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:32,241 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:34,231 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:04:37,023 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:38,233 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:42,025 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:43,234 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:47,026 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:48,235 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:52,027 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:53,236 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:04:57,028 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:04:58,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:02,029 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:05:03,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:04,232 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:05:07,030 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:05:09,235 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:12,031 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:05:14,236 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:17,032 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:05:19,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:20,556 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 02:05:20,559 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 02:05:20,561 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:05:20,567 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:05:21,134 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 02:05:24,569 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:47,162 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:47,164 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:47,165 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:47,164 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:05:47,165 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:47,381 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:05:50,167 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:05:55,168 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:01,501 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:01,532 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:04,233 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:06:05,235 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:07,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:10,236 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:12,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:15,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:17,247 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:20,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:22,248 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:25,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:27,249 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:30,240 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:32,250 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:34,234 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:06:36,235 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:37,251 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:41,236 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:42,252 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:46,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:47,253 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:51,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:52,254 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:06:56,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:06:57,255 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:01,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:02,256 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:04,235 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:07:07,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:07,257 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:12,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:12,258 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:17,240 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:17,259 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:22,240 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:22,260 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:27,241 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:27,261 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:32,242 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:32,262 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:34,236 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:07:37,263 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:38,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:42,263 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:43,238 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:47,264 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:48,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:52,265 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:53,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:07:57,266 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:07:58,240 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:02,267 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:08:03,241 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:04,165 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 02:08:04,170 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 02:08:04,171 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:08:04,176 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:08:04,236 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:08:04,534 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 02:08:09,237 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:36,561 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:36,563 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:36,563 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:36,564 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:36,564 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:36,565 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:08:36,826 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:08:39,566 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:44,567 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:49,568 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:50,457 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:08:54,568 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:08:56,068 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:08:59,569 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:01,070 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:04,238 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:09:05,239 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:06,071 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:10,240 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:11,072 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:15,241 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:16,073 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:20,241 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:21,074 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:25,242 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:26,075 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:30,243 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:31,076 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:34,241 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:09:36,077 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:36,242 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:41,078 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:41,243 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:46,083 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:46,244 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:51,084 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:51,244 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:09:56,085 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:09:56,245 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:01,086 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:01,246 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:04,239 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:10:06,087 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:07,241 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:11,088 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:12,242 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:15,267 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 02:10:15,269 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 02:10:15,269 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:15,278 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:15,898 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 02:10:16,089 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:17,280 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:22,281 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:27,282 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:27,477 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:32,283 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:40,003 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:40,004 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:10:40,011 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:43,007 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:44,253 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:44,255 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-21 02:10:44,271 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:44,276 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:44,285 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:44,287 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:44,296 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:44,298 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:44,307 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:44,309 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:44,317 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:44,319 DEBUG SenderThread:7197 [sender.py:send():378] send: history | |
| 2024-03-21 02:10:44,319 DEBUG SenderThread:7197 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-21 02:10:44,327 INFO SenderThread:7197 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-21 02:10:45,011 INFO Thread-12 :7197 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240321_015458-y11xrjxv/files/wandb-summary.json | |
| 2024-03-21 02:10:45,483 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:45,484 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:48,330 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:50,484 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:53,331 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:10:55,485 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:10:58,332 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:00,486 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:03,333 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:04,246 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:11:05,487 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:09,247 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:10,488 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:10,544 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: pause | |
| 2024-03-21 02:11:10,545 INFO HandlerThread:7197 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-03-21 02:11:10,545 INFO HandlerThread:7197 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-03-21 02:11:10,545 DEBUG SystemMonitor:7197 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-03-21 02:11:10,549 DEBUG SystemMonitor:7197 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-03-21 02:11:10,547 INFO HandlerThread:7197 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-03-21 02:11:10,554 INFO HandlerThread:7197 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-03-21 02:11:10,563 INFO HandlerThread:7197 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-03-21 02:11:10,563 INFO HandlerThread:7197 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-03-21 02:11:10,564 INFO HandlerThread:7197 [interfaces.py:finish():202] Joined network monitor | |
| 2024-03-21 02:11:10,564 DEBUG SenderThread:7197 [sender.py:send():378] send: stats | |
| 2024-03-21 02:11:14,565 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:15,489 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:19,566 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:20,490 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:24,567 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:25,491 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:29,568 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:30,492 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:34,569 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:35,493 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-03-21 02:11:38,632 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: resume | |
| 2024-03-21 02:11:38,633 INFO HandlerThread:7197 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-03-21 02:11:38,634 INFO HandlerThread:7197 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-21 02:11:38,634 INFO SystemMonitor:7197 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-21 02:11:38,637 INFO SystemMonitor:7197 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-21 02:11:38,640 INFO SystemMonitor:7197 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-21 02:11:38,642 INFO SystemMonitor:7197 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-21 02:11:38,649 INFO SystemMonitor:7197 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-21 02:11:38,651 INFO SystemMonitor:7197 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-21 02:11:39,570 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-21 02:11:40,615 DEBUG HandlerThread:7197 [handler.py:handle_request():146] handle_request: keepalive | |