| Use bfloat16 for core module. | |
| Use parallel attention and MLP in ViViT. | |
| Epoch 001/400 | |
| Train loss: 113924008.00 correlation: 0.0107 | |
| Validation loss: 200107760.00 correlation: 0.0263 | |
| Elapse: 605.46s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 002/400 | |
| Train loss: 97896768.00 correlation: 0.0340 | |
| Validation loss: 199235856.00 correlation: 0.0378 | |
| Elapse: 556.86s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 003/400 | |
| Train loss: 96883144.00 correlation: 0.0443 | |
| Validation loss: 198767200.00 correlation: 0.0405 | |
| Elapse: 560.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 004/400 | |
| Train loss: 96317000.00 correlation: 0.0499 | |
| Validation loss: 198194576.00 correlation: 0.0449 | |
| Elapse: 560.78s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 005/400 | |
| Train loss: 95592096.00 correlation: 0.0577 | |
| Validation loss: 197382576.00 correlation: 0.0493 | |
| Elapse: 563.47s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 006/400 | |
| Train loss: 94455696.00 correlation: 0.0692 | |
| Validation loss: 195917728.00 correlation: 0.0602 | |
| Elapse: 565.83s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 007/400 | |
| Train loss: 93257144.00 correlation: 0.0821 | |
| Validation loss: 193956304.00 correlation: 0.0721 | |
| Elapse: 566.60s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 008/400 | |
| Train loss: 92014728.00 correlation: 0.0951 | |
| Validation loss: 192039424.00 correlation: 0.0839 | |
| Elapse: 566.05s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 009/400 | |
| Train loss: 91067848.00 correlation: 0.1053 | |
| Validation loss: 190701360.00 correlation: 0.0935 | |
| Elapse: 563.36s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 010/400 | |
| Train loss: 90106464.00 correlation: 0.1149 | |
| Validation loss: 189259712.00 correlation: 0.1026 | |
| Elapse: 562.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 011/400 | |
| Train loss: 89183952.00 correlation: 0.1243 | |
| Validation loss: 187589920.00 correlation: 0.1130 | |
| Elapse: 560.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 012/400 | |
| Train loss: 88171952.00 correlation: 0.1348 | |
| Validation loss: 186131232.00 correlation: 0.1219 | |
| Elapse: 559.08s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 013/400 | |
| Train loss: 87220088.00 correlation: 0.1442 | |
| Validation loss: 184294144.00 correlation: 0.1328 | |
| Elapse: 564.57s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 014/400 | |
| Train loss: 86305888.00 correlation: 0.1535 | |
| Validation loss: 182766432.00 correlation: 0.1425 | |
| Elapse: 564.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 015/400 | |
| Train loss: 85306032.00 correlation: 0.1632 | |
| Validation loss: 181567952.00 correlation: 0.1505 | |
| Elapse: 562.48s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 016/400 | |
| Train loss: 84622736.00 correlation: 0.1701 | |
| Validation loss: 180351280.00 correlation: 0.1587 | |
| Elapse: 561.75s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 017/400 | |
| Train loss: 83839824.00 correlation: 0.1780 | |
| Validation loss: 179434864.00 correlation: 0.1644 | |
| Elapse: 561.90s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 018/400 | |
| Train loss: 83213000.00 correlation: 0.1840 | |
| Validation loss: 178816208.00 correlation: 0.1695 | |
| Elapse: 562.07s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 019/400 | |
| Train loss: 82673560.00 correlation: 0.1895 | |
| Validation loss: 178169568.00 correlation: 0.1740 | |
| Elapse: 562.05s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 020/400 | |
| Train loss: 82081376.00 correlation: 0.1949 | |
| Validation loss: 177435600.00 correlation: 0.1787 | |
| Elapse: 561.87s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 021/400 | |
| Train loss: 81644440.00 correlation: 0.1992 | |
| Validation loss: 176873408.00 correlation: 0.1824 | |
| Elapse: 562.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 022/400 | |
| Train loss: 81300944.00 correlation: 0.2027 | |
| Validation loss: 176298464.00 correlation: 0.1855 | |
| Elapse: 562.57s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 023/400 | |
| Train loss: 80864992.00 correlation: 0.2072 | |
| Validation loss: 175630304.00 correlation: 0.1905 | |
| Elapse: 562.55s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 024/400 | |
| Train loss: 80516536.00 correlation: 0.2102 | |
| Validation loss: 175099184.00 correlation: 0.1937 | |
| Elapse: 561.74s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 025/400 | |
| Train loss: 80141312.00 correlation: 0.2140 | |
| Validation loss: 174953952.00 correlation: 0.1954 | |
| Elapse: 563.63s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 026/400 | |
| Train loss: 79914320.00 correlation: 0.2162 | |
| Validation loss: 174397456.00 correlation: 0.1975 | |
| Elapse: 563.20s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 027/400 | |
| Train loss: 79520984.00 correlation: 0.2200 | |
| Validation loss: 174134560.00 correlation: 0.2014 | |
| Elapse: 562.71s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 028/400 | |
| Train loss: 79212208.00 correlation: 0.2229 | |
| Validation loss: 173471920.00 correlation: 0.2046 | |
| Elapse: 562.79s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 029/400 | |
| Train loss: 78973040.00 correlation: 0.2252 | |
| Validation loss: 173183520.00 correlation: 0.2064 | |
| Elapse: 563.08s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 030/400 | |
| Train loss: 78870784.00 correlation: 0.2264 | |
| Validation loss: 172775088.00 correlation: 0.2096 | |
| Elapse: 562.82s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 031/400 | |
| Train loss: 78490200.00 correlation: 0.2296 | |
| Validation loss: 172664992.00 correlation: 0.2107 | |
| Elapse: 563.05s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 032/400 | |
| Train loss: 78317200.00 correlation: 0.2318 | |
| Validation loss: 172019824.00 correlation: 0.2132 | |
| Elapse: 562.89s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 033/400 | |
| Train loss: 78137128.00 correlation: 0.2333 | |
| Validation loss: 171897024.00 correlation: 0.2150 | |
| Elapse: 562.71s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 034/400 | |
| Train loss: 77950032.00 correlation: 0.2354 | |
| Validation loss: 171802656.00 correlation: 0.2157 | |
| Elapse: 562.68s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 035/400 | |
| Train loss: 77766832.00 correlation: 0.2367 | |
| Validation loss: 171322160.00 correlation: 0.2180 | |
| Elapse: 561.92s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 036/400 | |
| Train loss: 77637216.00 correlation: 0.2384 | |
| Validation loss: 171034880.00 correlation: 0.2209 | |
| Elapse: 562.15s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 037/400 | |
| Train loss: 77482304.00 correlation: 0.2397 | |
| Validation loss: 171174608.00 correlation: 0.2203 | |
| Elapse: 564.93s | |
| Epoch 038/400 | |
| Train loss: 77429408.00 correlation: 0.2402 | |
| Validation loss: 170815888.00 correlation: 0.2218 | |
| Elapse: 564.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 039/400 | |
| Train loss: 77292792.00 correlation: 0.2412 | |
| Validation loss: 170692576.00 correlation: 0.2224 | |
| Elapse: 563.94s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 040/400 | |
| Train loss: 77122352.00 correlation: 0.2426 | |
| Validation loss: 170492064.00 correlation: 0.2256 | |
| Elapse: 565.03s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 041/400 | |
| Train loss: 77124456.00 correlation: 0.2430 | |
| Validation loss: 170532256.00 correlation: 0.2245 | |
| Elapse: 564.39s | |
| Epoch 042/400 | |
| Train loss: 76964592.00 correlation: 0.2445 | |
| Validation loss: 170406688.00 correlation: 0.2257 | |
| Elapse: 565.12s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 043/400 | |
| Train loss: 76869008.00 correlation: 0.2454 | |
| Validation loss: 170592080.00 correlation: 0.2248 | |
| Elapse: 563.98s | |
| Epoch 044/400 | |
| Train loss: 76798544.00 correlation: 0.2461 | |
| Validation loss: 170285664.00 correlation: 0.2253 | |
| Elapse: 564.88s | |
| Epoch 045/400 | |
| Train loss: 76706736.00 correlation: 0.2469 | |
| Validation loss: 170137504.00 correlation: 0.2276 | |
| Elapse: 564.78s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 046/400 | |
| Train loss: 76661312.00 correlation: 0.2474 | |
| Validation loss: 170203360.00 correlation: 0.2258 | |
| Elapse: 563.90s | |
| Epoch 047/400 | |
| Train loss: 76653904.00 correlation: 0.2479 | |
| Validation loss: 169920448.00 correlation: 0.2283 | |
| Elapse: 565.79s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 048/400 | |
| Train loss: 76510704.00 correlation: 0.2491 | |
| Validation loss: 169954384.00 correlation: 0.2279 | |
| Elapse: 566.41s | |
| Epoch 049/400 | |
| Train loss: 76493416.00 correlation: 0.2492 | |
| Validation loss: 169840224.00 correlation: 0.2290 | |
| Elapse: 566.13s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 050/400 | |
| Train loss: 76429016.00 correlation: 0.2495 | |
| Validation loss: 170036560.00 correlation: 0.2272 | |
| Elapse: 567.15s | |
| Epoch 051/400 | |
| Train loss: 76434184.00 correlation: 0.2500 | |
| Validation loss: 169668848.00 correlation: 0.2298 | |
| Elapse: 563.78s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 052/400 | |
| Train loss: 76368824.00 correlation: 0.2505 | |
| Validation loss: 169862640.00 correlation: 0.2292 | |
| Elapse: 563.52s | |
| Epoch 053/400 | |
| Train loss: 76235120.00 correlation: 0.2516 | |
| Validation loss: 169524560.00 correlation: 0.2315 | |
| Elapse: 565.24s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 054/400 | |
| Train loss: 76236176.00 correlation: 0.2518 | |
| Validation loss: 169235696.00 correlation: 0.2324 | |
| Elapse: 564.43s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 055/400 | |
| Train loss: 76272208.00 correlation: 0.2514 | |
| Validation loss: 169566032.00 correlation: 0.2303 | |
| Elapse: 564.62s | |
| Epoch 056/400 | |
| Train loss: 76095872.00 correlation: 0.2528 | |
| Validation loss: 169257008.00 correlation: 0.2329 | |
| Elapse: 564.52s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 057/400 | |
| Train loss: 76150160.00 correlation: 0.2523 | |
| Validation loss: 169306656.00 correlation: 0.2323 | |
| Elapse: 564.42s | |
| Epoch 058/400 | |
| Train loss: 75971168.00 correlation: 0.2540 | |
| Validation loss: 169281824.00 correlation: 0.2326 | |
| Elapse: 565.79s | |
| Epoch 059/400 | |
| Train loss: 76016632.00 correlation: 0.2537 | |
| Validation loss: 169175984.00 correlation: 0.2330 | |
| Elapse: 567.53s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 060/400 | |
| Train loss: 75961288.00 correlation: 0.2543 | |
| Validation loss: 169244832.00 correlation: 0.2312 | |
| Elapse: 564.74s | |
| Epoch 061/400 | |
| Train loss: 75918368.00 correlation: 0.2545 | |
| Validation loss: 169277088.00 correlation: 0.2310 | |
| Elapse: 564.73s | |
| Epoch 062/400 | |
| Train loss: 75869072.00 correlation: 0.2550 | |
| Validation loss: 169328944.00 correlation: 0.2320 | |
| Elapse: 564.39s | |
| Epoch 063/400 | |
| Train loss: 75762192.00 correlation: 0.2561 | |
| Validation loss: 169258112.00 correlation: 0.2316 | |
| Elapse: 564.19s | |
| Epoch 064/400 | |
| Train loss: 75825328.00 correlation: 0.2556 | |
| Validation loss: 169071840.00 correlation: 0.2332 | |
| Elapse: 564.93s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 065/400 | |
| Train loss: 75713256.00 correlation: 0.2566 | |
| Validation loss: 169214000.00 correlation: 0.2329 | |
| Elapse: 564.84s | |
| Epoch 066/400 | |
| Train loss: 75745160.00 correlation: 0.2561 | |
| Validation loss: 169041248.00 correlation: 0.2327 | |
| Elapse: 564.78s | |
| Epoch 067/400 | |
| Train loss: 75648152.00 correlation: 0.2572 | |
| Validation loss: 169051616.00 correlation: 0.2327 | |
| Elapse: 565.20s | |
| Epoch 068/400 | |
| Train loss: 75614368.00 correlation: 0.2577 | |
| Validation loss: 169221120.00 correlation: 0.2327 | |
| Elapse: 564.71s | |
| Epoch 069/400 | |
| Train loss: 75625776.00 correlation: 0.2574 | |
| Validation loss: 169122256.00 correlation: 0.2330 | |
| Elapse: 564.46s | |
| Loaded checkpoint from epoch 64 (correlation: 0.2332). | |
| Reduce learning rate of core to 1.4400e-03 (num. reduce: 1). | |
| Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1). | |
| Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1). | |
| Epoch 070/400 | |
| Train loss: 73759920.00 correlation: 0.2731 | |
| Validation loss: 167411552.00 correlation: 0.2444 | |
| Elapse: 565.27s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 071/400 | |
| Train loss: 73162624.00 correlation: 0.2785 | |
| Validation loss: 167276400.00 correlation: 0.2454 | |
| Elapse: 565.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 072/400 | |
| Train loss: 73062080.00 correlation: 0.2797 | |
| Validation loss: 167303840.00 correlation: 0.2453 | |
| Elapse: 565.70s | |
| Epoch 073/400 | |
| Train loss: 72859248.00 correlation: 0.2818 | |
| Validation loss: 167279840.00 correlation: 0.2448 | |
| Elapse: 566.21s | |
| Epoch 074/400 | |
| Train loss: 72867944.00 correlation: 0.2819 | |
| Validation loss: 167328304.00 correlation: 0.2445 | |
| Elapse: 566.10s | |
| Epoch 075/400 | |
| Train loss: 72806064.00 correlation: 0.2823 | |
| Validation loss: 167367392.00 correlation: 0.2443 | |
| Elapse: 566.39s | |
| Epoch 076/400 | |
| Train loss: 72707504.00 correlation: 0.2836 | |
| Validation loss: 167176992.00 correlation: 0.2450 | |
| Elapse: 566.85s | |
| Loaded checkpoint from epoch 71 (correlation: 0.2454). | |
| Reduce learning rate of core to 4.3200e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1). | |
| Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1). | |
| Epoch 077/400 | |
| Train loss: 72382144.00 correlation: 0.2856 | |
| Validation loss: 166842576.00 correlation: 0.2479 | |
| Elapse: 566.50s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 078/400 | |
| Train loss: 72078376.00 correlation: 0.2881 | |
| Validation loss: 166822496.00 correlation: 0.2483 | |
| Elapse: 567.42s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 079/400 | |
| Train loss: 71958248.00 correlation: 0.2896 | |
| Validation loss: 166738624.00 correlation: 0.2488 | |
| Elapse: 567.48s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 080/400 | |
| Train loss: 71895880.00 correlation: 0.2900 | |
| Validation loss: 166642640.00 correlation: 0.2490 | |
| Elapse: 567.47s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 081/400 | |
| Train loss: 71789800.00 correlation: 0.2911 | |
| Validation loss: 166705248.00 correlation: 0.2491 | |
| Elapse: 566.97s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 082/400 | |
| Train loss: 71770424.00 correlation: 0.2915 | |
| Validation loss: 166628368.00 correlation: 0.2487 | |
| Elapse: 567.08s | |
| Epoch 083/400 | |
| Train loss: 71742960.00 correlation: 0.2915 | |
| Validation loss: 166623744.00 correlation: 0.2491 | |
| Elapse: 567.32s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 084/400 | |
| Train loss: 71726080.00 correlation: 0.2915 | |
| Validation loss: 166593008.00 correlation: 0.2489 | |
| Elapse: 566.52s | |
| Epoch 085/400 | |
| Train loss: 71608472.00 correlation: 0.2930 | |
| Validation loss: 166582448.00 correlation: 0.2494 | |
| Elapse: 566.39s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 086/400 | |
| Train loss: 71622656.00 correlation: 0.2929 | |
| Validation loss: 166651488.00 correlation: 0.2488 | |
| Elapse: 567.27s | |
| Epoch 087/400 | |
| Train loss: 71550640.00 correlation: 0.2936 | |
| Validation loss: 166630768.00 correlation: 0.2490 | |
| Elapse: 566.70s | |
| Epoch 088/400 | |
| Train loss: 71491864.00 correlation: 0.2941 | |
| Validation loss: 166610080.00 correlation: 0.2493 | |
| Elapse: 567.65s | |
| Epoch 089/400 | |
| Train loss: 71458584.00 correlation: 0.2944 | |
| Validation loss: 166546464.00 correlation: 0.2492 | |
| Elapse: 566.68s | |
| Epoch 090/400 | |
| Train loss: 71462792.00 correlation: 0.2945 | |
| Validation loss: 166668320.00 correlation: 0.2492 | |
| Elapse: 566.84s | |
| Loaded checkpoint from epoch 85 (correlation: 0.2494). | |
| Reduce learning rate of core to 1.2960e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1). | |
| Epoch 091/400 | |
| Train loss: 71288968.00 correlation: 0.2953 | |
| Validation loss: 166494880.00 correlation: 0.2500 | |
| Elapse: 567.12s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 092/400 | |
| Train loss: 71275904.00 correlation: 0.2958 | |
| Validation loss: 166504256.00 correlation: 0.2498 | |
| Elapse: 566.80s | |
| Epoch 093/400 | |
| Train loss: 71229392.00 correlation: 0.2962 | |
| Validation loss: 166490080.00 correlation: 0.2502 | |
| Elapse: 567.91s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 094/400 | |
| Train loss: 71289360.00 correlation: 0.2955 | |
| Validation loss: 166481824.00 correlation: 0.2501 | |
| Elapse: 567.35s | |
| Epoch 095/400 | |
| Train loss: 71209928.00 correlation: 0.2962 | |
| Validation loss: 166463984.00 correlation: 0.2501 | |
| Elapse: 567.29s | |
| Epoch 096/400 | |
| Train loss: 71155936.00 correlation: 0.2969 | |
| Validation loss: 166447504.00 correlation: 0.2502 | |
| Elapse: 567.66s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 097/400 | |
| Train loss: 71163080.00 correlation: 0.2967 | |
| Validation loss: 166476448.00 correlation: 0.2501 | |
| Elapse: 567.23s | |
| Epoch 098/400 | |
| Train loss: 71090216.00 correlation: 0.2976 | |
| Validation loss: 166447952.00 correlation: 0.2502 | |
| Elapse: 567.15s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 099/400 | |
| Train loss: 71146160.00 correlation: 0.2971 | |
| Validation loss: 166436720.00 correlation: 0.2504 | |
| Elapse: 566.92s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 100/400 | |
| Train loss: 71064016.00 correlation: 0.2977 | |
| Validation loss: 166422944.00 correlation: 0.2502 | |
| Elapse: 567.03s | |
| Epoch 101/400 | |
| Train loss: 71056432.00 correlation: 0.2979 | |
| Validation loss: 166403840.00 correlation: 0.2505 | |
| Elapse: 567.52s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 102/400 | |
| Train loss: 71051576.00 correlation: 0.2978 | |
| Validation loss: 166443872.00 correlation: 0.2504 | |
| Elapse: 567.13s | |
| Epoch 103/400 | |
| Train loss: 71046608.00 correlation: 0.2979 | |
| Validation loss: 166479680.00 correlation: 0.2502 | |
| Elapse: 566.67s | |
| Epoch 104/400 | |
| Train loss: 71038120.00 correlation: 0.2981 | |
| Validation loss: 166454112.00 correlation: 0.2501 | |
| Elapse: 564.93s | |
| Epoch 105/400 | |
| Train loss: 70978192.00 correlation: 0.2988 | |
| Validation loss: 166414432.00 correlation: 0.2503 | |
| Elapse: 564.91s | |
| Epoch 106/400 | |
| Train loss: 70976960.00 correlation: 0.2990 | |
| Validation loss: 166474720.00 correlation: 0.2500 | |
| Elapse: 566.07s | |
| Loaded checkpoint from epoch 101 (correlation: 0.2505). | |
| Reduce learning rate of core to 3.8880e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1). | |
| Epoch 107/400 | |
| Train loss: 70972624.00 correlation: 0.2986 | |
| Validation loss: 166398144.00 correlation: 0.2506 | |
| Elapse: 566.75s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 108/400 | |
| Train loss: 70964176.00 correlation: 0.2987 | |
| Validation loss: 166382512.00 correlation: 0.2506 | |
| Elapse: 566.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 109/400 | |
| Train loss: 70976968.00 correlation: 0.2987 | |
| Validation loss: 166404432.00 correlation: 0.2505 | |
| Elapse: 567.32s | |
| Epoch 110/400 | |
| Train loss: 70951736.00 correlation: 0.2985 | |
| Validation loss: 166396896.00 correlation: 0.2506 | |
| Elapse: 567.77s | |
| Epoch 111/400 | |
| Train loss: 70895944.00 correlation: 0.2997 | |
| Validation loss: 166385856.00 correlation: 0.2506 | |
| Elapse: 566.89s | |
| Epoch 112/400 | |
| Train loss: 70880200.00 correlation: 0.2995 | |
| Validation loss: 166380672.00 correlation: 0.2507 | |
| Elapse: 566.94s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/002/ckpt/model_state.pt. | |
| Epoch 113/400 | |
| Train loss: 70861304.00 correlation: 0.2998 | |
| Validation loss: 166391184.00 correlation: 0.2506 | |
| Elapse: 566.54s | |
| Epoch 114/400 | |
| Train loss: 70921912.00 correlation: 0.2989 | |
| Validation loss: 166417776.00 correlation: 0.2504 | |
| Elapse: 567.26s | |
| Epoch 115/400 | |
| Train loss: 70914312.00 correlation: 0.2989 | |
| Validation loss: 166403376.00 correlation: 0.2506 | |
| Elapse: 566.92s | |
| Epoch 116/400 | |
| Train loss: 70903784.00 correlation: 0.2992 | |
| Validation loss: 166403360.00 correlation: 0.2506 | |
| Elapse: 567.11s | |
| Epoch 117/400 | |
| Train loss: 70866896.00 correlation: 0.2994 | |
| Validation loss: 166382688.00 correlation: 0.2505 | |
| Elapse: 566.94s | |
| Loaded checkpoint from epoch 112 (correlation: 0.2507). | |
| Reduce learning rate of core to 1.1664e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1). | |
| Epoch 118/400 | |
| Train loss: 70926000.00 correlation: 0.2989 | |
| Validation loss: 166380768.00 correlation: 0.2506 | |
| Elapse: 567.50s | |
| Epoch 119/400 | |
| Train loss: 70900808.00 correlation: 0.2993 | |
| Validation loss: 166386016.00 correlation: 0.2506 | |
| Elapse: 567.35s | |
| Epoch 120/400 | |
| Train loss: 70881328.00 correlation: 0.2993 | |
| Validation loss: 166387568.00 correlation: 0.2506 | |
| Elapse: 567.65s | |
| Epoch 121/400 | |
| Train loss: 70937256.00 correlation: 0.2984 | |
| Validation loss: 166393568.00 correlation: 0.2506 | |
| Elapse: 566.97s | |
| Epoch 122/400 | |
| Train loss: 70855888.00 correlation: 0.2996 | |
| Validation loss: 166378592.00 correlation: 0.2507 | |
| Elapse: 567.50s | |
| Loaded checkpoint from epoch 112 (correlation: 0.2507). | |
| Reduce learning rate of core to 3.4992e-06 (num. reduce: 2). | |
| Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 2). | |
| Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 2). | |
| Epoch 123/400 | |
| Train loss: 70938496.00 correlation: 0.2988 | |
| Validation loss: 166375792.00 correlation: 0.2507 | |
| Elapse: 567.50s | |
| Epoch 124/400 | |
| Train loss: 70966320.00 correlation: 0.2986 | |
| Validation loss: 166383664.00 correlation: 0.2506 | |
| Elapse: 567.77s | |
| Epoch 125/400 | |
| Train loss: 70872080.00 correlation: 0.2990 | |
| Validation loss: 166383328.00 correlation: 0.2506 | |
| Elapse: 567.87s | |
| Epoch 126/400 | |
| Train loss: 70866264.00 correlation: 0.2997 | |
| Validation loss: 166383136.00 correlation: 0.2506 | |
| Elapse: 567.21s | |
| Epoch 127/400 | |
| Train loss: 70836352.00 correlation: 0.2996 | |
| Validation loss: 166386816.00 correlation: 0.2506 | |
| Elapse: 566.90s | |
| Model has not improved after 2 LR reductions. | |
| Loaded checkpoint from epoch 112 (correlation: 0.2507). | |
| ValidationA: 0.2487 B: 0.2789 C: 0.2720 D: 0.2341 E: 0.2367 F: 0.2340 G: 0.2555 H: 0.2351 I: 0.2533 J: 0.2589 average: 0.2507 | |
| Results saved to /home/storage/runs/vivit_ensemble/002. | |