File size: 22,309 Bytes
3814f26
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
00948e4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
2024-03-22 04:34:07,566 INFO    StreamThr :4275 [internal.py:wandb_internal():86] W&B internal server running at pid: 4275, started at: 2024-03-22 04:34:07.562793
2024-03-22 04:34:07,589 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status
2024-03-22 04:34:12,594 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:12,594 INFO    WriterThread:4275 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/run-hize5f4p.wandb
2024-03-22 04:34:12,609 DEBUG   SenderThread:4275 [sender.py:send():378] send: header
2024-03-22 04:34:12,616 DEBUG   SenderThread:4275 [sender.py:send():378] send: run
2024-03-22 04:34:13,047 INFO    SenderThread:4275 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files
2024-03-22 04:34:13,047 INFO    SenderThread:4275 [sender.py:_start_run_threads():1099] run started: hize5f4p with start time 1711082047.565462
2024-03-22 04:34:13,064 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: check_version
2024-03-22 04:34:13,064 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: check_version
2024-03-22 04:34:13,182 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: run_start
2024-03-22 04:34:13,192 DEBUG   HandlerThread:4275 [system_info.py:__init__():26] System info init
2024-03-22 04:34:13,193 DEBUG   HandlerThread:4275 [system_info.py:__init__():41] System info init done
2024-03-22 04:34:13,193 INFO    HandlerThread:4275 [system_monitor.py:start():194] Starting system monitor
2024-03-22 04:34:13,193 INFO    SystemMonitor:4275 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-03-22 04:34:13,193 INFO    HandlerThread:4275 [system_monitor.py:probe():214] Collecting system info
2024-03-22 04:34:13,194 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started cpu monitoring
2024-03-22 04:34:13,195 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started disk monitoring
2024-03-22 04:34:13,196 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started gpu monitoring
2024-03-22 04:34:13,197 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started memory monitoring
2024-03-22 04:34:13,198 INFO    SystemMonitor:4275 [interfaces.py:start():190] Started network monitoring
2024-03-22 04:34:13,203 DEBUG   HandlerThread:4275 [system_info.py:probe():150] Probing system
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [gitlib.py:_init_repo():56] git repository is invalid
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [system_info.py:probe():198] Probing system done
2024-03-22 04:34:13,208 DEBUG   HandlerThread:4275 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-03-22T04:34:13.203984', 'startedAt': '2024-03-22T04:34:07.465777', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Mistral-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1yggNDPxtSX3bri4ON4ppYc-G-Q2zFtkQ', 'host': '6b98cdd15d10', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.876033782958984}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}}
2024-03-22 04:34:13,208 INFO    HandlerThread:4275 [system_monitor.py:probe():224] Finished collecting system info
2024-03-22 04:34:13,208 INFO    HandlerThread:4275 [system_monitor.py:probe():227] Publishing system info
2024-03-22 04:34:13,226 INFO    HandlerThread:4275 [system_monitor.py:probe():229] Finished publishing system info
2024-03-22 04:34:13,232 DEBUG   SenderThread:4275 [sender.py:send():378] send: files
2024-03-22 04:34:13,233 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now
2024-03-22 04:34:13,733 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: python_packages
2024-03-22 04:34:13,733 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: python_packages
2024-03-22 04:34:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:13,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:13,741 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:13,888 INFO    wandb-upload_0:4275 [upload_job.py:push():131] Uploaded file /tmp/tmp3i0vj0_swandb/55hqq2eg-wandb-metadata.json
2024-03-22 04:34:13,972 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:13,973 DEBUG   SenderThread:4275 [sender.py:send():378] send: config
2024-03-22 04:34:13,975 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:34:13,975 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:13,976 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:34:13,976 WARNING SenderThread:4275 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen)
2024-03-22 04:34:13,976 DEBUG   SenderThread:4275 [sender.py:send():378] send: telemetry
2024-03-22 04:34:14,052 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/requirements.txt
2024-03-22 04:34:14,053 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-metadata.json
2024-03-22 04:34:17,977 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:22,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:27,978 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:33,912 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:38,918 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:40,099 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml
2024-03-22 04:34:43,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:43,736 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:43,776 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:34:44,947 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:49,948 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:54,949 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:34:58,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:34:58,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:34:58,774 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:00,929 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:05,930 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:10,931 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:13,199 DEBUG   SystemMonitor:4275 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-03-22 04:35:13,200 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:35:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:13,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:13,774 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:16,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:21,922 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:26,923 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:28,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:28,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:31,951 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:36,952 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:41,952 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:43,201 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:35:43,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:43,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:43,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:35:47,914 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:52,914 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:57,915 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:35:58,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:35:58,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:35:58,775 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:02,958 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:07,958 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:12,959 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:13,202 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:36:13,734 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:13,736 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:13,776 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:18,925 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:23,926 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:28,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:28,735 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:28,777 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:28,959 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:33,960 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:38,377 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:36:38,384 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,385 DEBUG   SenderThread:4275 [sender.py:send():378] send: metric
2024-03-22 04:36:38,386 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:36:38,386 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:36:38,395 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:36:39,314 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:36:39,396 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:43,202 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:36:49,784 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:36:50,358 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/config.yaml
2024-03-22 04:36:50,399 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:36:50,401 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:36:50,402 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:36:55,622 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,735 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,752 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,753 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:14,753 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:14,753 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:37:14,754 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:14,783 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:37:16,284 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:21,289 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,399 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,407 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:31,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: internal_messages
2024-03-22 04:37:31,408 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:31,409 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:36,568 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:41,569 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:43,204 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:37:44,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:44,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:37:46,973 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:51,974 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:56,975 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:37:59,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:37:59,743 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:01,989 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:06,990 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:11,990 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:13,205 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:38:14,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:14,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:17,990 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:22,991 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:27,992 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:29,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:29,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:33,963 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:38,964 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:43,206 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:38:44,207 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:44,744 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:44,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:38:49,987 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:54,988 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:38:59,743 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:38:59,744 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:00,966 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:05,966 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:10,497 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: partial_history
2024-03-22 04:39:10,500 DEBUG   SenderThread:4275 [sender.py:send():378] send: history
2024-03-22 04:39:10,501 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: summary_record
2024-03-22 04:39:10,506 INFO    SenderThread:4275 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
2024-03-22 04:39:10,715 INFO    Thread-12 :4275 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240322_043407-hize5f4p/files/wandb-summary.json
2024-03-22 04:39:11,508 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:13,206 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:39:17,208 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:20,656 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:39:20,657 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:22,874 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:27,875 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:37,621 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:37,629 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: stop_status
2024-03-22 04:39:37,630 DEBUG   SenderThread:4275 [sender.py:send_request():405] send_request: stop_status
2024-03-22 04:39:38,874 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:43,207 DEBUG   SenderThread:4275 [sender.py:send():378] send: stats
2024-03-22 04:39:44,209 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report
2024-03-22 04:39:49,213 DEBUG   HandlerThread:4275 [handler.py:handle_request():146] handle_request: status_report