[[36m09-27 02:38:55[0m][[34mdora.distrib[0m][[32mINFO[0m] - world_size is 1, skipping init.[0m
[[36m09-27 02:38:55[0m][[34mflashy.solver[0m][[32mINFO[0m] - Instantiating solver MusicGenSolver for XP 7bb1d622[0m
[[36m09-27 02:38:55[0m][[34mflashy.solver[0m][[32mINFO[0m] - All XP logs are stored in /tmp/audiocraft_hari/xps/7bb1d622[0m
[[36m09-27 02:38:55[0m][[34maudiocraft.solvers.builders[0m][[32mINFO[0m] - Loading audio data split train: /kaggle/working/audiocraft/egs/folk[0m
[[36m09-27 02:38:55[0m][[34maudiocraft.solvers.builders[0m][[32mINFO[0m] - Loading audio data split valid: /kaggle/working/audiocraft/egs/folk[0m
[[36m09-27 02:38:55[0m][[34maudiocraft.solvers.builders[0m][[32mINFO[0m] - Loading audio data split evaluate: /kaggle/working/audiocraft/egs/folk[0m
[[36m09-27 02:38:55[0m][[34maudiocraft.solvers.builders[0m][[32mINFO[0m] - Loading audio data split generate: /kaggle/working/audiocraft/egs/folk[0m
[[36m09-27 02:38:55[0m][[34mroot[0m][[32mINFO[0m] - Getting pretrained compression model from HF facebook/encodec_32khz[0m
[[36m09-27 02:38:55[0m][[34mflashy.solver[0m][[32mINFO[0m] - Compression model has 4 codebooks with 2048 cardinality, and a framerate of 50[0m
[[36m09-27 02:38:55[0m][[34maudiocraft.modules.conditioners[0m][[32mINFO[0m] - T5 will be evaluated with autocast as float32[0m
[[36m09-27 02:39:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: 84d640e215de7863e944e465549d3e2e5faa07eb[0m
[[36m09-27 02:39:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - Initializing EMA on the model with decay = 0.99 every 10 updates[0m
[[36m09-27 02:39:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model size: 420.37 M params[0m
[[36m09-27 02:39:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - Base memory usage, with model, grad and optim: 6.73 GB[0m
[[36m09-27 02:39:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - Restoring weights and history.[0m
[[36m09-27 02:39:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - Loading a pretrained model. Ignoring 'load_best' and 'ignore_state_keys' params.[0m
[[36m09-27 02:39:02[0m][[34mflashy.solver[0m][[32mINFO[0m] - Checkpoint source is not the current xp: Load state_dict from best state.[0m
[[36m09-27 02:39:02[0m][[34mflashy.solver[0m][[32mINFO[0m] - Ignoring keys when loading best [][0m
[[36m09-27 02:39:02[0m][[34mflashy.solver[0m][[32mINFO[0m] - Loading state_dict from best state.[0m
[[36m09-27 02:39:04[0m][[34mflashy.solver[0m][[32mINFO[0m] - Re-initializing EMA from best state[0m
[[36m09-27 02:39:04[0m][[34mflashy.solver[0m][[32mINFO[0m] - Initializing EMA on the model with decay = 0.99 every 10 updates[0m
[[36m09-27 02:39:06[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: 776d041cbbcb8973c4968782a79f9bb63b53a727[0m
[[36m09-27 02:39:19[0m][[34maudiocraft.modules.codebooks_patterns[0m][[32mINFO[0m] - New pattern, time steps: 1500, sequence steps: 1504[0m
[[36m09-27 02:41:14[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 100/1000 | 0.79 it/sec | lr 9.55E-05 | grad_norm INF | grad_scale 18979.485 | ce 3.950 | ppl 62.856[0m
[[36m09-27 02:43:13[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 200/1000 | 0.82 it/sec | lr 9.99E-05 | grad_norm 7.454E+00 | grad_scale 16384.000 | ce 3.642 | ppl 44.336[0m
[[36m09-27 02:45:14[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 300/1000 | 0.82 it/sec | lr 9.99E-05 | grad_norm 5.754E+00 | grad_scale 16384.000 | ce 3.589 | ppl 39.746[0m
[[36m09-27 02:47:16[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 400/1000 | 0.82 it/sec | lr 9.97E-05 | grad_norm 5.457E+00 | grad_scale 16384.000 | ce 3.338 | ppl 32.399[0m
[[36m09-27 02:49:17[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 500/1000 | 0.82 it/sec | lr 9.95E-05 | grad_norm 4.997E+00 | grad_scale 16384.000 | ce 3.328 | ppl 31.657[0m
[[36m09-27 02:51:19[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 600/1000 | 0.82 it/sec | lr 9.93E-05 | grad_norm 4.529E+00 | grad_scale 16384.000 | ce 3.266 | ppl 29.985[0m
[[36m09-27 02:53:21[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 700/1000 | 0.82 it/sec | lr 9.90E-05 | grad_norm 4.046E+00 | grad_scale 16384.000 | ce 3.105 | ppl 25.236[0m
[[36m09-27 02:55:22[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 800/1000 | 0.82 it/sec | lr 9.86E-05 | grad_norm 3.947E+00 | grad_scale 16384.000 | ce 3.092 | ppl 25.148[0m
[[36m09-27 02:57:24[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 1 | 900/1000 | 0.82 it/sec | lr 9.83E-05 | grad_norm 3.829E+00 | grad_scale 16384.000 | ce 2.974 | ppl 23.287[0m
[[36m09-27 02:59:25[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mTrain Summary | Epoch 1 | lr=9.88E-05 | grad_norm=INF | grad_scale=16646.144 | ce=3.343 | ppl=33.996 | duration=1218.702[0m
[[36m09-27 02:59:38[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mValid Summary | Epoch 1 | ce=2.382 | ppl=10.829 | duration=12.130[0m
[[36m09-27 02:59:38[0m][[34mflashy.solver[0m][[32mINFO[0m] - New best state with ce=2.382 (was inf)[0m
[[36m09-27 02:59:43[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: fd1d2eb0e11786ece0dc7dcda0cff3b253ea5eb8[0m
[[36m09-27 03:00:07[0m][[34maudiocraft.utils.checkpoint[0m][[32mINFO[0m] - Checkpoint saved to /tmp/audiocraft_hari/xps/7bb1d622/checkpoint.th[0m
[[36m09-27 03:02:27[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 100/1000 | 0.72 it/sec | lr 9.73E-05 | grad_norm 3.541E+00 | grad_scale 16384.000 | ce 2.914 | ppl 21.070[0m
[[36m09-27 03:04:29[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 200/1000 | 0.77 it/sec | lr 9.68E-05 | grad_norm 3.519E+00 | grad_scale 16384.000 | ce 2.925 | ppl 20.493[0m
[[36m09-27 03:06:31[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 300/1000 | 0.78 it/sec | lr 9.62E-05 | grad_norm 3.297E+00 | grad_scale 16384.000 | ce 2.919 | ppl 20.219[0m
[[36m09-27 03:08:33[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 400/1000 | 0.79 it/sec | lr 9.56E-05 | grad_norm 3.206E+00 | grad_scale 16384.000 | ce 2.825 | ppl 19.182[0m
[[36m09-27 03:10:35[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 500/1000 | 0.80 it/sec | lr 9.49E-05 | grad_norm 3.196E+00 | grad_scale 16384.000 | ce 2.766 | ppl 18.441[0m
[[36m09-27 03:12:37[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 600/1000 | 0.80 it/sec | lr 9.42E-05 | grad_norm 3.375E+00 | grad_scale 16384.000 | ce 2.773 | ppl 18.401[0m
[[36m09-27 03:14:39[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 700/1000 | 0.80 it/sec | lr 9.35E-05 | grad_norm 2.782E+00 | grad_scale 16384.000 | ce 2.690 | ppl 16.668[0m
[[36m09-27 03:16:41[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 800/1000 | 0.81 it/sec | lr 9.27E-05 | grad_norm 3.032E+00 | grad_scale 16384.000 | ce 2.703 | ppl 16.660[0m
[[36m09-27 03:18:42[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 2 | 900/1000 | 0.81 it/sec | lr 9.18E-05 | grad_norm 2.772E+00 | grad_scale 16384.000 | ce 2.715 | ppl 16.596[0m
[[36m09-27 03:20:44[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mTrain Summary | Epoch 2 | lr=9.44E-05 | grad_norm=3.152E+00 | grad_scale=16384.000 | ce=2.787 | ppl=18.344 | duration=1236.408[0m
[[36m09-27 03:21:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mValid Summary | Epoch 2 | ce=2.074 | ppl=7.955 | duration=16.665[0m
[[36m09-27 03:21:01[0m][[34mflashy.solver[0m][[32mINFO[0m] - New best state with ce=2.074 (was 2.382)[0m
[[36m09-27 03:21:06[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: 103d6f3c0723fdd09761fde7bbb0a5b7746f0484[0m
[[36m09-27 03:21:38[0m][[34maudiocraft.utils.checkpoint[0m][[32mINFO[0m] - Checkpoint saved to /tmp/audiocraft_hari/xps/7bb1d622/checkpoint.th[0m
[[36m09-27 03:23:51[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 100/1000 | 0.76 it/sec | lr 9.00E-05 | grad_norm 2.612E+00 | grad_scale 31145.822 | ce 2.620 | ppl 15.199[0m
[[36m09-27 03:25:52[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 200/1000 | 0.79 it/sec | lr 8.91E-05 | grad_norm 2.765E+00 | grad_scale 32768.000 | ce 2.500 | ppl 13.496[0m
[[36m09-27 03:27:53[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 300/1000 | 0.80 it/sec | lr 8.81E-05 | grad_norm 2.642E+00 | grad_scale 32768.000 | ce 2.508 | ppl 13.660[0m
[[36m09-27 03:29:55[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 400/1000 | 0.81 it/sec | lr 8.70E-05 | grad_norm 2.541E+00 | grad_scale 32768.000 | ce 2.516 | ppl 13.777[0m
[[36m09-27 03:31:57[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 500/1000 | 0.81 it/sec | lr 8.60E-05 | grad_norm 2.610E+00 | grad_scale 32768.000 | ce 2.447 | ppl 13.087[0m
[[36m09-27 03:33:58[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 600/1000 | 0.81 it/sec | lr 8.49E-05 | grad_norm 2.530E+00 | grad_scale 32768.000 | ce 2.482 | ppl 13.249[0m
[[36m09-27 03:36:00[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 700/1000 | 0.81 it/sec | lr 8.37E-05 | grad_norm 2.555E+00 | grad_scale 32768.000 | ce 2.414 | ppl 12.265[0m
[[36m09-27 03:38:02[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 800/1000 | 0.81 it/sec | lr 8.25E-05 | grad_norm 2.527E+00 | grad_scale 32768.000 | ce 2.292 | ppl 11.530[0m
[[36m09-27 03:40:03[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 3 | 900/1000 | 0.82 it/sec | lr 8.13E-05 | grad_norm 2.528E+00 | grad_scale 32768.000 | ce 2.410 | ppl 12.200[0m
[[36m09-27 03:42:04[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mTrain Summary | Epoch 3 | lr=8.53E-05 | grad_norm=2.577E+00 | grad_scale=32604.160 | ce=2.457 | ppl=13.048 | duration=1225.959[0m
[[36m09-27 03:42:19[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mValid Summary | Epoch 3 | ce=1.858 | ppl=6.408 | duration=14.442[0m
[[36m09-27 03:42:19[0m][[34mflashy.solver[0m][[32mINFO[0m] - New best state with ce=1.858 (was 2.074)[0m
[[36m09-27 03:42:24[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: e94a3d0cd9b3c5ad12075f38aec0d365c79bad6b[0m
[[36m09-27 03:42:59[0m][[34maudiocraft.utils.checkpoint[0m][[32mINFO[0m] - Checkpoint saved to /tmp/audiocraft_hari/xps/7bb1d622/checkpoint.th[0m
[[36m09-27 03:45:16[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 100/1000 | 0.74 it/sec | lr 7.88E-05 | grad_norm 2.451E+00 | grad_scale 32768.000 | ce 2.343 | ppl 11.364[0m
[[36m09-27 03:47:18[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 200/1000 | 0.78 it/sec | lr 7.75E-05 | grad_norm 2.334E+00 | grad_scale 32768.000 | ce 2.288 | ppl 10.993[0m
[[36m09-27 03:49:20[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 300/1000 | 0.79 it/sec | lr 7.62E-05 | grad_norm 2.551E+00 | grad_scale 32768.000 | ce 2.305 | ppl 10.943[0m
[[36m09-27 03:51:22[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 400/1000 | 0.80 it/sec | lr 7.48E-05 | grad_norm 2.405E+00 | grad_scale 32768.000 | ce 2.310 | ppl 11.122[0m
[[36m09-27 03:53:24[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 500/1000 | 0.80 it/sec | lr 7.35E-05 | grad_norm 2.347E+00 | grad_scale 32768.000 | ce 2.235 | ppl 10.247[0m
[[36m09-27 03:55:25[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 600/1000 | 0.81 it/sec | lr 7.21E-05 | grad_norm 2.403E+00 | grad_scale 32768.000 | ce 2.120 | ppl 9.413[0m
[[36m09-27 03:57:27[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 700/1000 | 0.81 it/sec | lr 7.06E-05 | grad_norm 2.322E+00 | grad_scale 32768.000 | ce 2.270 | ppl 10.356[0m
[[36m09-27 03:59:28[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 800/1000 | 0.81 it/sec | lr 6.92E-05 | grad_norm 2.332E+00 | grad_scale 32768.000 | ce 2.106 | ppl 9.365[0m
[[36m09-27 04:01:30[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 4 | 900/1000 | 0.81 it/sec | lr 6.77E-05 | grad_norm 2.377E+00 | grad_scale 32768.000 | ce 2.280 | ppl 10.582[0m
[[36m09-27 04:03:31[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mTrain Summary | Epoch 4 | lr=7.27E-05 | grad_norm=2.390E+00 | grad_scale=32768.000 | ce=2.243 | ppl=10.386 | duration=1231.922[0m
[[36m09-27 04:03:44[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mValid Summary | Epoch 4 | ce=1.634 | ppl=5.123 | duration=12.240[0m
[[36m09-27 04:03:44[0m][[34mflashy.solver[0m][[32mINFO[0m] - New best state with ce=1.634 (was 1.858)[0m
[[36m09-27 04:03:49[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: c34af6581daf031d0ba4960e4aa56d6fad7c6e3f[0m
[[36m09-27 04:04:23[0m][[34maudiocraft.utils.checkpoint[0m][[32mINFO[0m] - Checkpoint saved to /tmp/audiocraft_hari/xps/7bb1d622/checkpoint.th[0m
[[36m09-27 04:06:37[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 100/1000 | 0.75 it/sec | lr 6.48E-05 | grad_norm 2.386E+00 | grad_scale 62291.644 | ce 2.196 | ppl 9.722[0m
[[36m09-27 04:08:37[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 200/1000 | 0.79 it/sec | lr 6.33E-05 | grad_norm 2.360E+00 | grad_scale 65536.000 | ce 2.119 | ppl 9.057[0m
[[36m09-27 04:10:38[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 300/1000 | 0.80 it/sec | lr 6.17E-05 | grad_norm 2.405E+00 | grad_scale 65536.000 | ce 2.152 | ppl 9.127[0m
[[36m09-27 04:12:40[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 400/1000 | 0.81 it/sec | lr 6.02E-05 | grad_norm 2.303E+00 | grad_scale 65536.000 | ce 2.095 | ppl 8.896[0m
[[36m09-27 04:14:42[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 500/1000 | 0.81 it/sec | lr 5.87E-05 | grad_norm 2.448E+00 | grad_scale 65536.000 | ce 2.060 | ppl 8.469[0m
[[36m09-27 04:16:44[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 600/1000 | 0.81 it/sec | lr 5.71E-05 | grad_norm 2.343E+00 | grad_scale 65536.000 | ce 2.107 | ppl 8.892[0m
[[36m09-27 04:18:45[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 700/1000 | 0.81 it/sec | lr 5.55E-05 | grad_norm 2.324E+00 | grad_scale 65536.000 | ce 1.968 | ppl 7.652[0m
[[36m09-27 04:20:47[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 800/1000 | 0.81 it/sec | lr 5.40E-05 | grad_norm 2.331E+00 | grad_scale 65536.000 | ce 1.950 | ppl 7.836[0m
[[36m09-27 04:22:49[0m][[34mflashy.solver[0m][[32mINFO[0m] - Train | Epoch 5 | 900/1000 | 0.81 it/sec | lr 5.24E-05 | grad_norm 2.265E+00 | grad_scale 65536.000 | ce 1.943 | ppl 7.613[0m
[[36m09-27 04:24:51[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mTrain Summary | Epoch 5 | lr=5.79E-05 | grad_norm=2.354E+00 | grad_scale=65208.320 | ce=2.058 | ppl=8.521 | duration=1227.401[0m
[[36m09-27 04:25:04[0m][[34mflashy.solver[0m][[32mINFO[0m] - [1mValid Summary | Epoch 5 | ce=1.479 | ppl=4.390 | duration=12.669[0m
[[36m09-27 04:25:04[0m][[34mflashy.solver[0m][[32mINFO[0m] - New best state with ce=1.479 (was 1.634)[0m
[[36m09-27 04:25:09[0m][[34mflashy.solver[0m][[32mINFO[0m] - Model hash: b426adae277f0c657cd94bd60e44014a5bca1936[0m
[[36m09-27 04:25:44[0m][[34maudiocraft.utils.checkpoint[0m][[32mINFO[0m] - Checkpoint saved to /tmp/audiocraft_hari/xps/7bb1d622/checkpoint.th[0m