new data

logs/main_log.txt

@@ -99904,3 +99904,50 @@ time (ms)
 [2021-09-28 06:32:50] PULSE: tr8-104B is scheduled to start in 17:29:26 (at 2021-09-29T00:02:17) (1277218 on 'gpu_p13' partition)
 [2021-09-28 06:32:50] PULSE: tr8-104B is waiting for the previous job to finish before scheduling a new one using the dependency mechanism (1277295_[1-10%1] on 'gpu_p13' partition)
 [2021-09-28 06:32:50] PULSE: tr8-104B is running for 12:49:24 since 2021-09-27T17:43:26 (1271196 on 'gpu_p13' partition (r7i7n[6-8],r8i0n[0-8],r8i1n[0-4],r8i7n[3-8],r9i0n[0-6,8],r9i1n[0-8],r9i2n0,r9i4n8,r9i5n[0-8],r9i6n[0-8],r9i7n[3-6])
+ iteration     8560/  159576 | consumed samples:       458160 | elapsed time per iteration (ms): 15526.6 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8570/  159576 | consumed samples:       459760 | elapsed time per iteration (ms): 15343.9 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8580/  159576 | consumed samples:       461360 | elapsed time per iteration (ms): 15516.0 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8590/  159576 | consumed samples:       462960 | elapsed time per iteration (ms): 15788.5 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8600/  159576 | consumed samples:       464560 | elapsed time per iteration (ms): 15421.5 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8610/  159576 | consumed samples:       466160 | elapsed time per iteration (ms): 15365.4 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8620/  159576 | consumed samples:       467760 | elapsed time per iteration (ms): 15460.6 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8630/  159576 | consumed samples:       469360 | elapsed time per iteration (ms): 15794.2 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8640/  159576 | consumed samples:       470960 | elapsed time per iteration (ms): 15928.5 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8650/  159576 | consumed samples:       472560 | elapsed time per iteration (ms): 15514.8 | learning rate: 6.000E-05 | global batch size:   160 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8660/  159576 | consumed samples:       474320 | elapsed time per iteration (ms): 16639.1 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8670/  159576 | consumed samples:       476080 | elapsed time per iteration (ms): 16569.6 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8680/  159576 | consumed samples:       477840 | elapsed time per iteration (ms): 16695.6 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8690/  159576 | consumed samples:       479600 | elapsed time per iteration (ms): 16700.3 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8700/  159576 | consumed samples:       481360 | elapsed time per iteration (ms): 16569.3 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8710/  159576 | consumed samples:       483120 | elapsed time per iteration (ms): 16526.6 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8720/  159576 | consumed samples:       484880 | elapsed time per iteration (ms): 16370.8 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8730/  159576 | consumed samples:       486640 | elapsed time per iteration (ms): 16678.1 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8740/  159576 | consumed samples:       488400 | elapsed time per iteration (ms): 16715.4 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8750/  159576 | consumed samples:       490160 | elapsed time per iteration (ms): 16605.2 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8760/  159576 | consumed samples:       491920 | elapsed time per iteration (ms): 16522.8 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     8770/  159576 | consumed samples:       493680 | elapsed time per iteration (ms): 16607.3 | learning rate: 6.000E-05 | global batch size:   176 | loss scale: 1.0 | grad norm: 5533.127 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+[2021-09-28 07:32:48] PULSE: tr8-104B is scheduled to start in 17:38:05 (at 2021-09-29T01:10:54) (1277218 on 'gpu_p13' partition)
+[2021-09-28 07:32:48] PULSE: tr8-104B is waiting for the previous job to finish before scheduling a new one using the dependency mechanism (1277295_[1-10%1] on 'gpu_p13' partition)
+[2021-09-28 07:32:48] PULSE: tr8-104B is running for 13:49:22 since 2021-09-27T17:43:26 (1271196 on 'gpu_p13' partition (r7i7n[6-8],r8i0n[0-8],r8i1n[0-4],r8i7n[3-8],r9i0n[0-6,8],r9i1n[0-8],r9i2n0,r9i4n8,r9i5n[0-8],r9i6n[0-8],r9i7n[3-6])