bigscience
/

tr8-104B-logs

TensorBoard

Model card Files Files and versions Metrics Training metrics Community

bigscience-bot commited on Sep 30, 2021

Commit

0292bfa

•

1 Parent(s): f8573dc

new data

Browse files

Files changed (1) hide show

logs/main_log.txt +166 -0

logs/main_log.txt CHANGED Viewed

@@ -110912,3 +110912,169 @@ time (ms)
 [Rank 193] (after 6220 iterations) memory (MB) | allocated: 5861.55029296875 | max allocated: 10754.46533203125 | reserved: 19054.0 | max reserved: 19054.0
  iteration     6230/  159576 | consumed samples:       195200 | elapsed time per iteration (ms): 29715.9 | learning rate: 5.400E-05 | global batch size:    80 | lm loss: 6.325600E+00 | loss scale: 4096.0 | grad norm: 93189.900 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 time (ms)

 [Rank 193] (after 6220 iterations) memory (MB) | allocated: 5861.55029296875 | max allocated: 10754.46533203125 | reserved: 19054.0 | max reserved: 19054.0
  iteration     6230/  159576 | consumed samples:       195200 | elapsed time per iteration (ms): 29715.9 | learning rate: 5.400E-05 | global batch size:    80 | lm loss: 6.325600E+00 | loss scale: 4096.0 | grad norm: 93189.900 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 time (ms)
+ iteration     6240/  159576 | consumed samples:       196000 | elapsed time per iteration (ms): 29850.6 | learning rate: 5.423E-05 | global batch size:    80 | lm loss: 6.314528E+00 | loss scale: 4096.0 | grad norm: 153013.405 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.154311
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.693372
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.853277
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.842906
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.156230
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.007043
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.141515
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.998821
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.978375
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.153948
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.046725
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.105339
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.839469
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.886734
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.846006
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.129153
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.156799
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.917236
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.199060
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.036142
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.135256
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.068995
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.104105
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.945723
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.807536
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.878620
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.885677
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.967674
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.949363
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.167258
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.095681
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.905876
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.070411
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.200017
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.000211
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.878945
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:10.053487
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.966731
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.867150
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:11:21 CEST)" was missed by 0:00:09.998094
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.832527
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.900956
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.959521
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.693701
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.740910
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.700204
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.008585
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.547640
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.707561
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.053319
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.861273
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.995798
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.054171
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.853107
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.890343
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.821832
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.989445
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.008173
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.923175
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.983410
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.011032
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.771456
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.732863
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.010526
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.854397
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.760050
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.803591
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.949886
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:09.021500
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.958312
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.799978
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.820922
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.661789
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.697242
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.739874
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.924615
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.907646
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.733121
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.721307
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:12:21 CEST)" was missed by 0:00:08.852267
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.436041
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.611707
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.526661
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.563051
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.297217
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.344454
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.303732
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.612125
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.614514
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.151174
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.311093
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.656833
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.614044
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.464781
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.599319
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.657678
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.456642
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.493848
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.363548
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.425338
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.528064
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.592991
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.553354
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.624952
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.561765
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.504510
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.403512
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.424427
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.586926
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.265296
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.300740
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.375030
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.336389
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.343398
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.457902
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.511153
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.407115
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.336631
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.324806
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:13:21 CEST)" was missed by 0:00:07.455805
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.805411
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.673073
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.981482
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.520498
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.680458
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:05.026200
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.834157
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.968697
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:05.027056
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.825991
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.863229
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.794725
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.962332
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.981080
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.896079
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.922735
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.994312
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.873866
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.932435
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.666589
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.713819
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.793773
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.956314
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.983905
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.634635
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.670132
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.744374
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.983402
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.827251
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.732936
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.897469
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.776498
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.931203
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.772888
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.705977
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.694159
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.705781
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.712784
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.880530
+WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 04:14:21 CEST)" was missed by 0:00:04.825162
+ iteration     6250/  159576 | consumed samples:       196800 | elapsed time per iteration (ms): 29009.9 | learning rate: 5.445E-05 | global batch size:    80 | lm loss: 6.303601E+00 | loss scale: 4096.0 | grad norm: 137433.627 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration     6260/  159576 | consumed samples:       197600 | elapsed time per iteration (ms): 28924.6 | learning rate: 5.467E-05 | global batch size:    80 | lm loss: 6.323338E+00 | loss scale: 4096.0 | grad norm: 108774.777 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)