new data

logs/main_log.txt

@@ -86978,3 +86978,52 @@ time (ms)
  iteration    10090/  159576 | consumed samples:       784912 | elapsed time per iteration (ms): 15060.9 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 time (ms)
 [2021-09-27 05:14:32] PULSE: tr8-104B is running for 1:20:16 since 2021-09-27T03:54:16 (1188168 on 'gpu_p13' partition (r6i5n[7-8],r6i6n0,r6i7n[7-8],r7i0n[0-5],r7i1n[7-8],r7i2n[0-1,5,8],r7i3n2,r7i5n7,r7i6n[1-4,8],r7i7n[0-4,6-8],r8i0n[0-8],r8i1n[0-4],r8i2n8,r8i3n[0-3,8],r8i4n[0-1],r8i6n[2-3,5-6],r8i7n[3-8],r9i0n[0-6,8],r9i1n[0-8],r9i2n[0,3-8],r9i3n[0-2,6-8],r9i4n[0-6,8],r9i5n[0-8],r9i6n[0-8],r9i7n[1-8])
+ iteration    10100/  159576 | consumed samples:       787632 | elapsed time per iteration (ms): 14624.0 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10110/  159576 | consumed samples:       790352 | elapsed time per iteration (ms): 14621.7 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10120/  159576 | consumed samples:       793072 | elapsed time per iteration (ms): 14685.1 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10130/  159576 | consumed samples:       795792 | elapsed time per iteration (ms): 14531.8 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10140/  159576 | consumed samples:       798512 | elapsed time per iteration (ms): 14629.6 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10150/  159576 | consumed samples:       801232 | elapsed time per iteration (ms): 14771.8 | learning rate: 6.000E-05 | global batch size:   272 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10160/  159576 | consumed samples:       803984 | elapsed time per iteration (ms): 14889.9 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10170/  159576 | consumed samples:       806864 | elapsed time per iteration (ms): 15471.9 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10180/  159576 | consumed samples:       809744 | elapsed time per iteration (ms): 15228.6 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10190/  159576 | consumed samples:       812624 | elapsed time per iteration (ms): 15425.1 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10200/  159576 | consumed samples:       815504 | elapsed time per iteration (ms): 15390.8 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10210/  159576 | consumed samples:       818384 | elapsed time per iteration (ms): 15293.9 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10220/  159576 | consumed samples:       821264 | elapsed time per iteration (ms): 15259.9 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10230/  159576 | consumed samples:       824144 | elapsed time per iteration (ms): 15547.4 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10240/  159576 | consumed samples:       827024 | elapsed time per iteration (ms): 15375.5 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10250/  159576 | consumed samples:       829904 | elapsed time per iteration (ms): 15322.8 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10260/  159576 | consumed samples:       832784 | elapsed time per iteration (ms): 15280.3 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10270/  159576 | consumed samples:       835664 | elapsed time per iteration (ms): 15390.4 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10280/  159576 | consumed samples:       838544 | elapsed time per iteration (ms): 15339.6 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10290/  159576 | consumed samples:       841424 | elapsed time per iteration (ms): 15252.5 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10300/  159576 | consumed samples:       844304 | elapsed time per iteration (ms): 15146.5 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10310/  159576 | consumed samples:       847184 | elapsed time per iteration (ms): 15389.7 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10320/  159576 | consumed samples:       850064 | elapsed time per iteration (ms): 15348.5 | learning rate: 6.000E-05 | global batch size:   288 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    10330/  159576 | consumed samples:       853072 | elapsed time per iteration (ms): 15779.0 | learning rate: 6.000E-05 | global batch size:   304 | loss scale: 1.0 | grad norm: 0.000 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+[2021-09-27 06:14:35] PULSE: tr8-104B is running for 2:20:19 since 2021-09-27T03:54:16 (1188168 on 'gpu_p13' partition (r6i5n[7-8],r6i6n0,r6i7n[7-8],r7i0n[0-5],r7i1n[7-8],r7i2n[0-1,5,8],r7i3n2,r7i5n7,r7i6n[1-4,8],r7i7n[0-4,6-8],r8i0n[0-8],r8i1n[0-4],r8i2n8,r8i3n[0-3,8],r8i4n[0-1],r8i6n[2-3,5-6],r8i7n[3-8],r9i0n[0-6,8],r9i1n[0-8],r9i2n[0,3-8],r9i3n[0-2,6-8],r9i4n[0-6,8],r9i5n[0-8],r9i6n[0-8],r9i7n[1-8])