| 2024-03-22 04:34:07,566 INFO StreamThr :4275 [internal.py:wandb_internal():86] W&B internal server running at pid: 4275, started at: 2024-03-22 04:34:07.562793 | |
| 2024-03-22 04:34:07,589 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status | |
| 2024-03-22 04:34:12,594 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:12,594 INFO WriterThread:4275 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/run-hize5f4p.wandb | |
| 2024-03-22 04:34:12,609 DEBUG SenderThread:4275 [sender.py:send():378] send: header | |
| 2024-03-22 04:34:12,616 DEBUG SenderThread:4275 [sender.py:send():378] send: run | |
| 2024-03-22 04:34:13,047 INFO SenderThread:4275 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files | |
| 2024-03-22 04:34:13,047 INFO SenderThread:4275 [sender.py:_start_run_threads():1099] run started: hize5f4p with start time 1711082047.565462 | |
| 2024-03-22 04:34:13,064 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-03-22 04:34:13,064 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: check_version | |
| 2024-03-22 04:34:13,182 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-03-22 04:34:13,192 DEBUG HandlerThread:4275 [system_info.py:__init__():26] System info init | |
| 2024-03-22 04:34:13,193 DEBUG HandlerThread:4275 [system_info.py:__init__():41] System info init done | |
| 2024-03-22 04:34:13,193 INFO HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor | |
| 2024-03-22 04:34:13,193 INFO SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-03-22 04:34:13,193 INFO HandlerThread:4275 [system_monitor.py:probe():214] Collecting system info | |
| 2024-03-22 04:34:13,194 INFO SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-03-22 04:34:13,195 INFO SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring | |
| 2024-03-22 04:34:13,196 INFO SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-03-22 04:34:13,197 INFO SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring | |
| 2024-03-22 04:34:13,198 INFO SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring | |
| 2024-03-22 04:34:13,203 DEBUG HandlerThread:4275 [system_info.py:probe():150] Probing system | |
| 2024-03-22 04:34:13,208 DEBUG HandlerThread:4275 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-03-22 04:34:13,208 DEBUG HandlerThread:4275 [system_info.py:probe():198] Probing system done | |
| 2024-03-22 04:34:13,208 DEBUG HandlerThread:4275 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-22T04:34:13.203984', 'startedAt': '2024-03-22T04:34:07.465777', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1yggNDPxtSX3bri4ON4ppYc-G-Q2zFtkQ', 'host': '6b98cdd15d10', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.876033782958984}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}} | |
| 2024-03-22 04:34:13,208 INFO HandlerThread:4275 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-03-22 04:34:13,208 INFO HandlerThread:4275 [system_monitor.py:probe():227] Publishing system info | |
| 2024-03-22 04:34:13,226 INFO HandlerThread:4275 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-03-22 04:34:13,232 DEBUG SenderThread:4275 [sender.py:send():378] send: files | |
| 2024-03-22 04:34:13,233 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-03-22 04:34:13,733 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-03-22 04:34:13,733 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: python_packages | |
| 2024-03-22 04:34:13,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:13,735 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:34:13,741 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:13,888 INFO wandb-upload_0:4275 [upload_job.py:push():131] Uploaded file /tmp/tmp3i0vj0_swandb/55hqq2eg-wandb-metadata.json | |
| 2024-03-22 04:34:13,972 DEBUG SenderThread:4275 [sender.py:send():378] send: telemetry | |
| 2024-03-22 04:34:13,973 DEBUG SenderThread:4275 [sender.py:send():378] send: config | |
| 2024-03-22 04:34:13,975 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:34:13,975 DEBUG SenderThread:4275 [sender.py:send():378] send: telemetry | |
| 2024-03-22 04:34:13,976 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:34:13,976 WARNING SenderThread:4275 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-03-22 04:34:13,976 DEBUG SenderThread:4275 [sender.py:send():378] send: telemetry | |
| 2024-03-22 04:34:14,052 INFO Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/requirements.txt | |
| 2024-03-22 04:34:14,053 INFO Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-metadata.json | |
| 2024-03-22 04:34:17,977 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:22,978 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:27,978 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:28,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:28,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:34:28,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:33,912 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:38,918 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:40,099 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml | |
| 2024-03-22 04:34:43,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:43,736 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:43,776 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:34:44,947 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:49,948 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:54,949 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:34:58,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:34:58,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:34:58,774 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:00,929 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:05,930 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:10,931 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:13,199 DEBUG SystemMonitor:4275 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-03-22 04:35:13,200 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:35:13,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:13,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:13,774 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:16,922 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:21,922 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:26,923 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:28,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:28,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:28,775 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:31,951 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:36,952 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:41,952 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:43,201 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:35:43,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:43,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:43,775 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:35:47,914 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:52,914 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:57,915 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:35:58,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:35:58,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:35:58,775 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:02,958 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:07,958 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:12,959 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:13,202 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:36:13,734 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:36:13,736 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:36:13,776 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:18,925 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:23,926 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:28,735 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:36:28,735 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:36:28,777 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:28,959 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:33,960 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:38,377 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-03-22 04:36:38,384 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,385 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,385 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,385 DEBUG SenderThread:4275 [sender.py:send():378] send: metric | |
| 2024-03-22 04:36:38,386 DEBUG SenderThread:4275 [sender.py:send():378] send: history | |
| 2024-03-22 04:36:38,386 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: summary_record | |
| 2024-03-22 04:36:38,395 INFO SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-03-22 04:36:39,314 INFO Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json | |
| 2024-03-22 04:36:39,396 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:43,202 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:36:49,784 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:36:50,358 INFO Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml | |
| 2024-03-22 04:36:50,399 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:36:50,401 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:36:50,402 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:36:55,622 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,735 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,752 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,753 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:14,753 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:14,753 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:37:14,754 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:37:14,783 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:37:16,284 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:21,289 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:31,399 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:31,407 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:31,408 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-03-22 04:37:31,408 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:31,409 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:37:36,568 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:41,569 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:43,204 DEBUG SenderThread:4275 [sender.py:send():378] send: stats | |
| 2024-03-22 04:37:44,743 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-03-22 04:37:44,744 DEBUG SenderThread:4275 [sender.py:send_request():405] send_request: stop_status | |
| 2024-03-22 04:37:46,973 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:51,974 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-03-22 04:37:56,975 DEBUG HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report | |