Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-ssft

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
sft
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
0ae2f86
•
1 Parent(s): d893097

Training in progress, step 900

Browse files
Files changed (7) hide show
  1. model.safetensors +1 -1
  2. runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 +2 -2
  3. wandb/debug-internal.log +201 -0
  4. wandb/run-20240201_173828-py26nu6m/files/output.log +86 -0
  5. wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json +1 -1
  6. wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log +201 -0
  7. wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb +0 -0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1eecd214ab022ddfc50f0784a4b3b2b8acaf582c0333ff0b484c54baad9db149
3
  size 3141646744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b92d043a09b94d7a22c478b47119a6bad9b5e914d6e9d6e6a566e2c9cc0a2be
3
  size 3141646744
runs/Feb01_17-38-02_ip-26-0-165-24/events.out.tfevents.1706809106.ip-26-0-165-24.237059.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:67160b39a55667879606165c5a2b8098ed407a589d90b08020bea7ca7b53f98c
3
- size 29574
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:150e10d36280c0b789cf11077e52e9dfc0006692f1a394589a62f320ce274b43
3
+ size 32714
wandb/debug-internal.log CHANGED
@@ -1732,3 +1732,204 @@
1732
  2024-02-01 17:53:50,981 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1733
  2024-02-01 17:53:50,981 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1734
  2024-02-01 17:53:51,163 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1732
  2024-02-01 17:53:50,981 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1733
  2024-02-01 17:53:50,981 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1734
  2024-02-01 17:53:51,163 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1735
+ 2024-02-01 17:53:53,166 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1736
+ 2024-02-01 17:53:53,931 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1737
+ 2024-02-01 17:53:53,931 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1738
+ 2024-02-01 17:53:53,933 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1739
+ 2024-02-01 17:53:53,933 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1740
+ 2024-02-01 17:53:53,935 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1741
+ 2024-02-01 17:53:54,168 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1742
+ 2024-02-01 17:53:55,169 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1743
+ 2024-02-01 17:53:57,172 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1744
+ 2024-02-01 17:53:58,550 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1745
+ 2024-02-01 17:53:58,552 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1746
+ 2024-02-01 17:53:58,552 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1747
+ 2024-02-01 17:53:58,554 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1748
+ 2024-02-01 17:53:58,955 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1749
+ 2024-02-01 17:53:58,956 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1750
+ 2024-02-01 17:53:59,176 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1751
+ 2024-02-01 17:53:59,176 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1752
+ 2024-02-01 17:54:01,178 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1753
+ 2024-02-01 17:54:03,181 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1754
+ 2024-02-01 17:54:03,184 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1755
+ 2024-02-01 17:54:03,186 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1756
+ 2024-02-01 17:54:03,186 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1757
+ 2024-02-01 17:54:03,188 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1758
+ 2024-02-01 17:54:04,111 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1759
+ 2024-02-01 17:54:04,183 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1760
+ 2024-02-01 17:54:05,185 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1761
+ 2024-02-01 17:54:06,000 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1762
+ 2024-02-01 17:54:06,002 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1763
+ 2024-02-01 17:54:06,002 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1764
+ 2024-02-01 17:54:07,187 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1765
+ 2024-02-01 17:54:07,821 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1766
+ 2024-02-01 17:54:07,822 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1767
+ 2024-02-01 17:54:07,822 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1768
+ 2024-02-01 17:54:07,824 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1769
+ 2024-02-01 17:54:08,189 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1770
+ 2024-02-01 17:54:09,191 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1771
+ 2024-02-01 17:54:09,669 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1772
+ 2024-02-01 17:54:11,193 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1773
+ 2024-02-01 17:54:12,451 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1774
+ 2024-02-01 17:54:12,452 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1775
+ 2024-02-01 17:54:12,453 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1776
+ 2024-02-01 17:54:12,454 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1777
+ 2024-02-01 17:54:13,197 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1778
+ 2024-02-01 17:54:13,197 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1779
+ 2024-02-01 17:54:15,199 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1780
+ 2024-02-01 17:54:15,220 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1781
+ 2024-02-01 17:54:17,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1782
+ 2024-02-01 17:54:17,074 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1783
+ 2024-02-01 17:54:17,074 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1784
+ 2024-02-01 17:54:17,076 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1785
+ 2024-02-01 17:54:17,203 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1786
+ 2024-02-01 17:54:17,203 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1787
+ 2024-02-01 17:54:19,205 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1788
+ 2024-02-01 17:54:20,771 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1789
+ 2024-02-01 17:54:20,991 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1790
+ 2024-02-01 17:54:21,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1791
+ 2024-02-01 17:54:21,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1792
+ 2024-02-01 17:54:21,208 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1793
+ 2024-02-01 17:54:21,970 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1794
+ 2024-02-01 17:54:21,971 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1795
+ 2024-02-01 17:54:21,972 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1796
+ 2024-02-01 17:54:21,973 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1797
+ 2024-02-01 17:54:22,210 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1798
+ 2024-02-01 17:54:23,212 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1799
+ 2024-02-01 17:54:25,214 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1800
+ 2024-02-01 17:54:26,601 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1801
+ 2024-02-01 17:54:26,602 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1802
+ 2024-02-01 17:54:26,603 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1803
+ 2024-02-01 17:54:26,604 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1804
+ 2024-02-01 17:54:26,606 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1805
+ 2024-02-01 17:54:27,218 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1806
+ 2024-02-01 17:54:27,218 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1807
+ 2024-02-01 17:54:28,957 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1808
+ 2024-02-01 17:54:29,220 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1809
+ 2024-02-01 17:54:31,791 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1810
+ 2024-02-01 17:54:33,226 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1811
+ 2024-02-01 17:54:35,991 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1812
+ 2024-02-01 17:54:36,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1813
+ 2024-02-01 17:54:36,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1814
+ 2024-02-01 17:54:37,143 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1815
+ 2024-02-01 17:54:37,231 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1816
+ 2024-02-01 17:54:39,233 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1817
+ 2024-02-01 17:54:42,405 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1818
+ 2024-02-01 17:54:43,238 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1819
+ 2024-02-01 17:54:45,241 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1820
+ 2024-02-01 17:54:46,097 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1821
+ 2024-02-01 17:54:46,099 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1822
+ 2024-02-01 17:54:46,099 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1823
+ 2024-02-01 17:54:46,101 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1824
+ 2024-02-01 17:54:46,243 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1825
+ 2024-02-01 17:54:47,245 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1826
+ 2024-02-01 17:54:47,939 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1827
+ 2024-02-01 17:54:49,247 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1828
+ 2024-02-01 17:54:50,708 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1829
+ 2024-02-01 17:54:50,709 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1830
+ 2024-02-01 17:54:50,710 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1831
+ 2024-02-01 17:54:50,712 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1832
+ 2024-02-01 17:54:50,992 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1833
+ 2024-02-01 17:54:51,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1834
+ 2024-02-01 17:54:51,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1835
+ 2024-02-01 17:54:51,251 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1836
+ 2024-02-01 17:54:51,251 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1837
+ 2024-02-01 17:54:53,254 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1838
+ 2024-02-01 17:54:53,475 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1839
+ 2024-02-01 17:54:55,256 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1840
+ 2024-02-01 17:54:55,324 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1841
+ 2024-02-01 17:54:55,325 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1842
+ 2024-02-01 17:54:55,326 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1843
+ 2024-02-01 17:54:55,328 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1844
+ 2024-02-01 17:54:56,259 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1845
+ 2024-02-01 17:54:57,260 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1846
+ 2024-02-01 17:54:58,974 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1847
+ 2024-02-01 17:54:58,975 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1848
+ 2024-02-01 17:54:59,263 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1849
+ 2024-02-01 17:54:59,946 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1850
+ 2024-02-01 17:54:59,947 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1851
+ 2024-02-01 17:54:59,947 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1852
+ 2024-02-01 17:54:59,949 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1853
+ 2024-02-01 17:55:00,265 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1854
+ 2024-02-01 17:55:01,266 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1855
+ 2024-02-01 17:55:03,269 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1856
+ 2024-02-01 17:55:04,577 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1857
+ 2024-02-01 17:55:04,578 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1858
+ 2024-02-01 17:55:04,580 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1859
+ 2024-02-01 17:55:04,580 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1860
+ 2024-02-01 17:55:04,582 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1861
+ 2024-02-01 17:55:05,273 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1862
+ 2024-02-01 17:55:05,273 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1863
+ 2024-02-01 17:55:05,991 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1864
+ 2024-02-01 17:55:06,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1865
+ 2024-02-01 17:55:06,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1866
+ 2024-02-01 17:55:07,275 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1867
+ 2024-02-01 17:55:09,205 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1868
+ 2024-02-01 17:55:09,206 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1869
+ 2024-02-01 17:55:09,206 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1870
+ 2024-02-01 17:55:09,208 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1871
+ 2024-02-01 17:55:09,279 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1872
+ 2024-02-01 17:55:09,279 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1873
+ 2024-02-01 17:55:10,127 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1874
+ 2024-02-01 17:55:11,282 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1875
+ 2024-02-01 17:55:13,284 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1876
+ 2024-02-01 17:55:13,834 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1877
+ 2024-02-01 17:55:13,836 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1878
+ 2024-02-01 17:55:13,836 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1879
+ 2024-02-01 17:55:13,838 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1880
+ 2024-02-01 17:55:14,287 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1881
+ 2024-02-01 17:55:15,288 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1882
+ 2024-02-01 17:55:15,685 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1883
+ 2024-02-01 17:55:17,291 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1884
+ 2024-02-01 17:55:18,463 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1885
+ 2024-02-01 17:55:18,464 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1886
+ 2024-02-01 17:55:18,464 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1887
+ 2024-02-01 17:55:18,466 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1888
+ 2024-02-01 17:55:19,294 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1889
+ 2024-02-01 17:55:19,295 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1890
+ 2024-02-01 17:55:20,992 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1891
+ 2024-02-01 17:55:21,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1892
+ 2024-02-01 17:55:21,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1893
+ 2024-02-01 17:55:21,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1894
+ 2024-02-01 17:55:21,297 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1895
+ 2024-02-01 17:55:23,104 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1896
+ 2024-02-01 17:55:23,105 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1897
+ 2024-02-01 17:55:23,106 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1898
+ 2024-02-01 17:55:23,108 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1899
+ 2024-02-01 17:55:23,301 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1900
+ 2024-02-01 17:55:23,301 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1901
+ 2024-02-01 17:55:24,302 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1902
+ 2024-02-01 17:55:26,821 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1903
+ 2024-02-01 17:55:27,306 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1904
+ 2024-02-01 17:55:27,752 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1905
+ 2024-02-01 17:55:27,753 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1906
+ 2024-02-01 17:55:27,754 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1907
+ 2024-02-01 17:55:27,755 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1908
+ 2024-02-01 17:55:28,309 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1909
+ 2024-02-01 17:55:28,310 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1910
+ 2024-02-01 17:55:28,961 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1911
+ 2024-02-01 17:55:31,313 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1912
+ 2024-02-01 17:55:32,374 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1913
+ 2024-02-01 17:55:32,375 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1914
+ 2024-02-01 17:55:32,376 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1915
+ 2024-02-01 17:55:32,376 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1916
+ 2024-02-01 17:55:32,378 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1917
+ 2024-02-01 17:55:33,316 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1918
+ 2024-02-01 17:55:33,317 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1919
+ 2024-02-01 17:55:34,318 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1920
+ 2024-02-01 17:55:35,992 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1921
+ 2024-02-01 17:55:36,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1922
+ 2024-02-01 17:55:36,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1923
+ 2024-02-01 17:55:37,011 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1924
+ 2024-02-01 17:55:37,012 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1925
+ 2024-02-01 17:55:37,012 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1926
+ 2024-02-01 17:55:37,014 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1927
+ 2024-02-01 17:55:37,323 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1928
+ 2024-02-01 17:55:37,323 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1929
+ 2024-02-01 17:55:37,940 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1930
+ 2024-02-01 17:55:38,324 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1931
+ 2024-02-01 17:55:41,328 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1932
+ 2024-02-01 17:55:41,644 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1933
+ 2024-02-01 17:55:41,645 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1934
+ 2024-02-01 17:55:41,645 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1935
+ 2024-02-01 17:55:41,647 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
wandb/run-20240201_173828-py26nu6m/files/output.log CHANGED
@@ -714,3 +714,89 @@
714
  79%|███████████████████████████████████████████████████████████████ | 859/1090 [15:13<03:34, 1.08it/s]
715
 
716
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
714
  79%|███████████████████████████████████████████████████████████████ | 859/1090 [15:13<03:34, 1.08it/s]
715
 
716
 
717
+
718
+ 79%|███████████████████████████████████████████████████████████████▍ | 865/1090 [15:19<03:28, 1.08it/s]
719
+
720
+
721
+ 80%|███████████████████████████████████████████████████████████████▊ | 870/1090 [15:24<03:23, 1.08it/s]
722
+
723
+
724
+ 80%|████████████████████████████████████████████████████████████████▏ | 874/1090 [15:27<03:20, 1.08it/s]
725
+
726
+
727
+
728
+ 81%|████████████████████████████████████████████████████████████████▌ | 880/1090 [15:33<03:14, 1.08it/s]
729
+
730
+
731
+ 81%|████████████████████████████████████████████████████████████████▉ | 885/1090 [15:37<03:09, 1.08it/s]
732
+
733
+
734
+ 82%|█████████████████████████████████████████████████████████████████▏ | 889/1090 [15:41<03:05, 1.08it/s]
735
+
736
+
737
+
738
+ 82%|█████████████████████████████████████████████████████████████████▋ | 895/1090 [15:47<03:16, 1.01s/it]
739
+
740
+
741
+ 82%|█████████████████████████████████████████████████████████████████▉ | 899/1090 [15:51<03:00, 1.06it/s]
742
+ 83%|██████████████████████████████████████████████████████████████████ | 900/1090 [15:52<02:58, 1.06it/s][INFO|trainer.py:2889] 2024-02-01 17:54:27,664 >> Saving model checkpoint to ./tmp-checkpoint-900
743
+ [INFO|configuration_utils.py:483] 2024-02-01 17:54:27,667 >> Configuration saved in ./tmp-checkpoint-900/config.json
744
+ [INFO|configuration_utils.py:594] 2024-02-01 17:54:27,670 >> Configuration saved in ./tmp-checkpoint-900/generation_config.json
745
+ [2024-02-01 17:54:30,775] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step900 is about to be saved!
746
+ [2024-02-01 17:54:30,780] [INFO] [logging.py:96:log_dist] [Rank 0] Saving model checkpoint: ./tmp-checkpoint-900/global_step900/zero_pp_rank_0_mp_rank_00_model_states.pt
747
+ [2024-02-01 17:54:30,780] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-900/global_step900/zero_pp_rank_0_mp_rank_00_model_states.pt...
748
+ [2024-02-01 17:54:30,784] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-900/global_step900/zero_pp_rank_0_mp_rank_00_model_states.pt.
749
+ [2024-02-01 17:54:30,790] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving ./tmp-checkpoint-900/global_step900/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
750
+ [INFO|modeling_utils.py:2382] 2024-02-01 17:54:30,743 >> Model weights saved in ./tmp-checkpoint-900/pytorch_model.bin
751
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:54:30,746 >> tokenizer config file saved in ./tmp-checkpoint-900/tokenizer_config.json
752
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:54:30,749 >> Special tokens file saved in ./tmp-checkpoint-900/special_tokens_map.json
753
+ /fsx/sanchit/miniconda3/envs/venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1879: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.
754
+ warnings.warn(
755
+ [2024-02-01 17:54:34,521] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved ./tmp-checkpoint-900/global_step900/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
756
+ [2024-02-01 17:54:34,525] [INFO] [engine.py:3393:_save_zero_checkpoint] zero checkpoint saved ./tmp-checkpoint-900/global_step900/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
757
+ [2024-02-01 17:54:34,855] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step900 is ready now!
758
+ [INFO|tokenization_utils_base.py:2432] 2024-02-01 17:54:37,837 >> tokenizer config file saved in ./tokenizer_config.json
759
+ [INFO|tokenization_utils_base.py:2441] 2024-02-01 17:54:37,839 >> Special tokens file saved in ./special_tokens_map.json
760
+ [INFO|trainer.py:2979] 2024-02-01 17:54:37,868 >> Deleting older checkpoint [checkpoint-800] due to args.save_total_limit
761
+
762
+
763
+ 83%|██████████████████████████████████████████████████████████████████▍ | 905/1090 [16:11<06:09, 2.00s/it]
764
+
765
+
766
+ 83%|██████████████████████████████████████████████████████████████████▋ | 909/1090 [16:15<03:33, 1.18s/it]
767
+
768
+
769
+ 84%|███████████████████████████████████████████████████████████████████ | 914/1090 [16:19<02:50, 1.03it/s]
770
+
771
+
772
+
773
+ 84%|███████████████████████████████████████████████████████████████████▌ | 920/1090 [16:25<02:37, 1.08it/s]
774
+
775
+
776
+ 85%|███████████████████████████████████████████████████████████████████▉ | 925/1090 [16:30<02:33, 1.08it/s]
777
+
778
+
779
+ 85%|████████████████████████████████████████████████████████████████████▏ | 929/1090 [16:33<02:28, 1.08it/s]
780
+
781
+
782
+
783
+ 86%|████████████████████████████████████████████████████████████████████▌ | 935/1090 [16:39<02:23, 1.08it/s]
784
+
785
+
786
+ 86%|████████████████████████████████████████████████████████████████████▉ | 940/1090 [16:43<02:18, 1.08it/s]
787
+
788
+
789
+ 87%|█████████████████████████████████████████████████████████████████████▎ | 944/1090 [16:47<02:15, 1.08it/s]
790
+
791
+
792
+ 87%|█████████████████████████████████████████████████████████████████████▌ | 948/1090 [16:51<02:12, 1.08it/s]
793
+
794
+
795
+
796
+ 88%|██████████████████████████████████████████████████████████████████████ | 955/1090 [16:57<02:04, 1.08it/s]
797
+
798
+
799
+ 88%|██████████████████████████████████████████████████████████████████████▍ | 959/1090 [17:01<02:01, 1.08it/s]
800
+
801
+
802
+ 88%|██████████████████████████████████████████████████████████████████████▋ | 963/1090 [17:05<01:57, 1.08it/s]
wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 2.4713, "train/learning_rate": 2.592169155352031e-06, "train/epoch": 0.79, "train/global_step": 860, "_timestamp": 1706810029.288467, "_runtime": 920.836592912674, "_step": 172}
 
1
+ {"train/loss": 2.475, "train/learning_rate": 7.905777244947954e-07, "train/epoch": 0.89, "train/global_step": 965, "_timestamp": 1706810141.6438708, "_runtime": 1033.1919968128204, "_step": 193}
wandb/run-20240201_173828-py26nu6m/logs/debug-internal.log CHANGED
@@ -1732,3 +1732,204 @@
1732
  2024-02-01 17:53:50,981 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1733
  2024-02-01 17:53:50,981 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1734
  2024-02-01 17:53:51,163 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1732
  2024-02-01 17:53:50,981 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1733
  2024-02-01 17:53:50,981 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1734
  2024-02-01 17:53:51,163 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1735
+ 2024-02-01 17:53:53,166 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1736
+ 2024-02-01 17:53:53,931 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1737
+ 2024-02-01 17:53:53,931 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1738
+ 2024-02-01 17:53:53,933 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1739
+ 2024-02-01 17:53:53,933 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1740
+ 2024-02-01 17:53:53,935 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1741
+ 2024-02-01 17:53:54,168 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1742
+ 2024-02-01 17:53:55,169 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1743
+ 2024-02-01 17:53:57,172 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1744
+ 2024-02-01 17:53:58,550 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1745
+ 2024-02-01 17:53:58,552 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1746
+ 2024-02-01 17:53:58,552 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1747
+ 2024-02-01 17:53:58,554 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1748
+ 2024-02-01 17:53:58,955 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1749
+ 2024-02-01 17:53:58,956 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1750
+ 2024-02-01 17:53:59,176 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1751
+ 2024-02-01 17:53:59,176 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1752
+ 2024-02-01 17:54:01,178 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1753
+ 2024-02-01 17:54:03,181 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1754
+ 2024-02-01 17:54:03,184 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1755
+ 2024-02-01 17:54:03,186 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1756
+ 2024-02-01 17:54:03,186 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1757
+ 2024-02-01 17:54:03,188 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1758
+ 2024-02-01 17:54:04,111 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1759
+ 2024-02-01 17:54:04,183 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1760
+ 2024-02-01 17:54:05,185 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1761
+ 2024-02-01 17:54:06,000 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1762
+ 2024-02-01 17:54:06,002 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1763
+ 2024-02-01 17:54:06,002 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1764
+ 2024-02-01 17:54:07,187 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1765
+ 2024-02-01 17:54:07,821 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1766
+ 2024-02-01 17:54:07,822 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1767
+ 2024-02-01 17:54:07,822 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1768
+ 2024-02-01 17:54:07,824 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1769
+ 2024-02-01 17:54:08,189 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1770
+ 2024-02-01 17:54:09,191 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1771
+ 2024-02-01 17:54:09,669 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1772
+ 2024-02-01 17:54:11,193 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1773
+ 2024-02-01 17:54:12,451 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1774
+ 2024-02-01 17:54:12,452 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1775
+ 2024-02-01 17:54:12,453 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1776
+ 2024-02-01 17:54:12,454 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1777
+ 2024-02-01 17:54:13,197 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1778
+ 2024-02-01 17:54:13,197 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1779
+ 2024-02-01 17:54:15,199 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1780
+ 2024-02-01 17:54:15,220 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1781
+ 2024-02-01 17:54:17,072 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1782
+ 2024-02-01 17:54:17,074 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1783
+ 2024-02-01 17:54:17,074 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1784
+ 2024-02-01 17:54:17,076 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1785
+ 2024-02-01 17:54:17,203 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1786
+ 2024-02-01 17:54:17,203 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1787
+ 2024-02-01 17:54:19,205 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1788
+ 2024-02-01 17:54:20,771 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1789
+ 2024-02-01 17:54:20,991 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1790
+ 2024-02-01 17:54:21,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1791
+ 2024-02-01 17:54:21,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1792
+ 2024-02-01 17:54:21,208 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1793
+ 2024-02-01 17:54:21,970 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1794
+ 2024-02-01 17:54:21,971 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1795
+ 2024-02-01 17:54:21,972 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1796
+ 2024-02-01 17:54:21,973 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1797
+ 2024-02-01 17:54:22,210 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1798
+ 2024-02-01 17:54:23,212 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1799
+ 2024-02-01 17:54:25,214 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1800
+ 2024-02-01 17:54:26,601 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1801
+ 2024-02-01 17:54:26,602 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1802
+ 2024-02-01 17:54:26,603 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1803
+ 2024-02-01 17:54:26,604 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1804
+ 2024-02-01 17:54:26,606 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1805
+ 2024-02-01 17:54:27,218 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1806
+ 2024-02-01 17:54:27,218 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1807
+ 2024-02-01 17:54:28,957 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1808
+ 2024-02-01 17:54:29,220 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1809
+ 2024-02-01 17:54:31,791 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1810
+ 2024-02-01 17:54:33,226 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1811
+ 2024-02-01 17:54:35,991 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1812
+ 2024-02-01 17:54:36,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1813
+ 2024-02-01 17:54:36,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1814
+ 2024-02-01 17:54:37,143 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1815
+ 2024-02-01 17:54:37,231 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1816
+ 2024-02-01 17:54:39,233 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1817
+ 2024-02-01 17:54:42,405 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1818
+ 2024-02-01 17:54:43,238 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1819
+ 2024-02-01 17:54:45,241 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1820
+ 2024-02-01 17:54:46,097 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1821
+ 2024-02-01 17:54:46,099 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1822
+ 2024-02-01 17:54:46,099 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1823
+ 2024-02-01 17:54:46,101 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1824
+ 2024-02-01 17:54:46,243 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1825
+ 2024-02-01 17:54:47,245 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1826
+ 2024-02-01 17:54:47,939 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1827
+ 2024-02-01 17:54:49,247 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1828
+ 2024-02-01 17:54:50,708 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1829
+ 2024-02-01 17:54:50,709 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1830
+ 2024-02-01 17:54:50,710 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1831
+ 2024-02-01 17:54:50,712 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1832
+ 2024-02-01 17:54:50,992 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1833
+ 2024-02-01 17:54:51,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1834
+ 2024-02-01 17:54:51,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1835
+ 2024-02-01 17:54:51,251 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1836
+ 2024-02-01 17:54:51,251 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1837
+ 2024-02-01 17:54:53,254 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1838
+ 2024-02-01 17:54:53,475 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1839
+ 2024-02-01 17:54:55,256 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1840
+ 2024-02-01 17:54:55,324 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1841
+ 2024-02-01 17:54:55,325 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1842
+ 2024-02-01 17:54:55,326 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1843
+ 2024-02-01 17:54:55,328 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1844
+ 2024-02-01 17:54:56,259 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1845
+ 2024-02-01 17:54:57,260 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1846
+ 2024-02-01 17:54:58,974 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1847
+ 2024-02-01 17:54:58,975 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1848
+ 2024-02-01 17:54:59,263 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1849
+ 2024-02-01 17:54:59,946 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1850
+ 2024-02-01 17:54:59,947 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1851
+ 2024-02-01 17:54:59,947 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1852
+ 2024-02-01 17:54:59,949 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1853
+ 2024-02-01 17:55:00,265 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1854
+ 2024-02-01 17:55:01,266 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1855
+ 2024-02-01 17:55:03,269 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1856
+ 2024-02-01 17:55:04,577 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1857
+ 2024-02-01 17:55:04,578 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1858
+ 2024-02-01 17:55:04,580 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1859
+ 2024-02-01 17:55:04,580 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1860
+ 2024-02-01 17:55:04,582 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1861
+ 2024-02-01 17:55:05,273 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1862
+ 2024-02-01 17:55:05,273 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1863
+ 2024-02-01 17:55:05,991 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1864
+ 2024-02-01 17:55:06,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1865
+ 2024-02-01 17:55:06,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1866
+ 2024-02-01 17:55:07,275 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1867
+ 2024-02-01 17:55:09,205 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1868
+ 2024-02-01 17:55:09,206 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1869
+ 2024-02-01 17:55:09,206 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1870
+ 2024-02-01 17:55:09,208 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1871
+ 2024-02-01 17:55:09,279 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1872
+ 2024-02-01 17:55:09,279 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1873
+ 2024-02-01 17:55:10,127 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1874
+ 2024-02-01 17:55:11,282 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1875
+ 2024-02-01 17:55:13,284 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1876
+ 2024-02-01 17:55:13,834 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1877
+ 2024-02-01 17:55:13,836 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1878
+ 2024-02-01 17:55:13,836 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1879
+ 2024-02-01 17:55:13,838 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1880
+ 2024-02-01 17:55:14,287 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1881
+ 2024-02-01 17:55:15,288 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1882
+ 2024-02-01 17:55:15,685 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1883
+ 2024-02-01 17:55:17,291 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1884
+ 2024-02-01 17:55:18,463 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1885
+ 2024-02-01 17:55:18,464 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1886
+ 2024-02-01 17:55:18,464 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1887
+ 2024-02-01 17:55:18,466 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1888
+ 2024-02-01 17:55:19,294 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1889
+ 2024-02-01 17:55:19,295 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1890
+ 2024-02-01 17:55:20,992 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1891
+ 2024-02-01 17:55:21,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1892
+ 2024-02-01 17:55:21,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1893
+ 2024-02-01 17:55:21,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1894
+ 2024-02-01 17:55:21,297 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1895
+ 2024-02-01 17:55:23,104 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1896
+ 2024-02-01 17:55:23,105 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1897
+ 2024-02-01 17:55:23,106 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1898
+ 2024-02-01 17:55:23,108 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1899
+ 2024-02-01 17:55:23,301 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1900
+ 2024-02-01 17:55:23,301 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1901
+ 2024-02-01 17:55:24,302 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1902
+ 2024-02-01 17:55:26,821 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1903
+ 2024-02-01 17:55:27,306 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1904
+ 2024-02-01 17:55:27,752 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1905
+ 2024-02-01 17:55:27,753 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1906
+ 2024-02-01 17:55:27,754 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1907
+ 2024-02-01 17:55:27,755 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1908
+ 2024-02-01 17:55:28,309 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1909
+ 2024-02-01 17:55:28,310 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1910
+ 2024-02-01 17:55:28,961 DEBUG SenderThread:237521 [sender.py:send():382] send: stats
1911
+ 2024-02-01 17:55:31,313 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1912
+ 2024-02-01 17:55:32,374 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1913
+ 2024-02-01 17:55:32,375 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1914
+ 2024-02-01 17:55:32,376 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1915
+ 2024-02-01 17:55:32,376 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1916
+ 2024-02-01 17:55:32,378 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1917
+ 2024-02-01 17:55:33,316 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1918
+ 2024-02-01 17:55:33,317 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1919
+ 2024-02-01 17:55:34,318 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1920
+ 2024-02-01 17:55:35,992 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: internal_messages
1921
+ 2024-02-01 17:55:36,001 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: stop_status
1922
+ 2024-02-01 17:55:36,001 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: stop_status
1923
+ 2024-02-01 17:55:37,011 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1924
+ 2024-02-01 17:55:37,012 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1925
+ 2024-02-01 17:55:37,012 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1926
+ 2024-02-01 17:55:37,014 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
1927
+ 2024-02-01 17:55:37,323 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/wandb-summary.json
1928
+ 2024-02-01 17:55:37,323 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1929
+ 2024-02-01 17:55:37,940 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: status_report
1930
+ 2024-02-01 17:55:38,324 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1931
+ 2024-02-01 17:55:41,328 INFO Thread-12 :237521 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-ssft/wandb/run-20240201_173828-py26nu6m/files/output.log
1932
+ 2024-02-01 17:55:41,644 DEBUG HandlerThread:237521 [handler.py:handle_request():146] handle_request: partial_history
1933
+ 2024-02-01 17:55:41,645 DEBUG SenderThread:237521 [sender.py:send():382] send: history
1934
+ 2024-02-01 17:55:41,645 DEBUG SenderThread:237521 [sender.py:send_request():409] send_request: summary_record
1935
+ 2024-02-01 17:55:41,647 INFO SenderThread:237521 [sender.py:_save_file():1392] saving file wandb-summary.json with policy end
wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb CHANGED
Binary files a/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb and b/wandb/run-20240201_173828-py26nu6m/run-py26nu6m.wandb differ