Hugging Face's logo

bigscience
/
tr8-104B-logs

TensorBoard
Model card Files Metrics Community
bigscience-bot commited on
Commit
f56a13f
•
1 Parent(s): 710d236

new data

Browse files
Files changed (1) hide show
  1. logs/main_log.txt +265 -0
logs/main_log.txt CHANGED
@@ -112378,3 +112378,268 @@ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._mea
112378
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:42:21 CEST)" was missed by 0:00:06.777713
112379
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:42:21 CEST)" was missed by 0:00:06.521077
112380
  [2021-09-30 06:42:15] PULSE: tr8-104B is running for 2:50:08 since 2021-09-30T03:52:07 (1289770 on 'gpu_p13' partition (r6i4n[5-6,8],r6i5n[4-5],r7i0n[5-8],r7i1n0,r8i2n8,r8i4n1,r8i7n[3-8],r9i0n[0-8],r9i1n[0-8],r9i2n[3-8],r9i3n[7-8],r9i4n[0-2],r9i5n[2,5-7],r9i6n[2-8],r14i7n[1-6])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
112378
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:42:21 CEST)" was missed by 0:00:06.777713
112379
  WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:42:21 CEST)" was missed by 0:00:06.521077
112380
  [2021-09-30 06:42:15] PULSE: tr8-104B is running for 2:50:08 since 2021-09-30T03:52:07 (1289770 on 'gpu_p13' partition (r6i4n[5-6,8],r6i5n[4-5],r7i0n[5-8],r7i1n0,r8i2n8,r8i4n1,r8i7n[3-8],r9i0n[0-8],r9i1n[0-8],r9i2n[3-8],r9i3n[7-8],r9i4n[0-2],r9i5n[2,5-7],r9i6n[2-8],r14i7n[1-6])
112381
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.661339
112382
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.767808
112383
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.819301
112384
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.861590
112385
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.804060
112386
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.862410
112387
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.662600
112388
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.698576
112389
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.568258
112390
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.640896
112391
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.630060
112392
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.731476
112393
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.829712
112394
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.502001
112395
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.608311
112396
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.505518
112397
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.548100
112398
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.669567
112399
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.816541
112400
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.758153
112401
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.766550
112402
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.709320
112403
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.508511
112404
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.629207
112405
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.791725
112406
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.816985
112407
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.356025
112408
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.541348
112409
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.470092
112410
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.579756
112411
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.732875
112412
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.797822
112413
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.715899
112414
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.611889
112415
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.549298
112416
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.515939
112417
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.541203
112418
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.818876
112419
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.529570
112420
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:43:21 CEST)" was missed by 0:00:06.660572
112421
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.185299
112422
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.243652
112423
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.042616
112424
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.149108
112425
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.200577
112426
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.242847
112427
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.043914
112428
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.079867
112429
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.949531
112430
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.022166
112431
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.011341
112432
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.197798
112433
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.090584
112434
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.883295
112435
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.989590
112436
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.737295
112437
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.851317
112438
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.961040
112439
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.929405
112440
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.050849
112441
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.993154
112442
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.112804
112443
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.211059
112444
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.147850
112445
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.930571
112446
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.172989
112447
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.922626
112448
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.886820
112449
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.114170
112450
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.179112
112451
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.139470
112452
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.889840
112453
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.010516
112454
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.198262
112455
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.922478
112456
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.097194
112457
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.897236
112458
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.200171
112459
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:06.910868
112460
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:44:21 CEST)" was missed by 0:00:07.041867
112461
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.710719
112462
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.478403
112463
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.616193
112464
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.456627
112465
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.318412
112466
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.428087
112467
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.709967
112468
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.652435
112469
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.510964
112470
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.509725
112471
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.546961
112472
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.416597
112473
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.489271
112474
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.664863
112475
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.678108
112476
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.614875
112477
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.557685
112478
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.640092
112479
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.667685
112480
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.204374
112481
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.389672
112482
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.396487
112483
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.517938
112484
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.581198
112485
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.564232
112486
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.460224
112487
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.579878
112488
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.606548
112489
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.350402
112490
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.397660
112491
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.356901
112492
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.477602
112493
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.665346
112494
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.353901
112495
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.389568
112496
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.646178
112497
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.364301
112498
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.377887
112499
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.667240
112500
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:45:21 CEST)" was missed by 0:00:07.508910
112501
+ iteration 6560/ 159576 | consumed samples: 221600 | elapsed time per iteration (ms): 30277.7 | learning rate: 6.000E-05 | global batch size: 80 | lm loss: 6.306966E+00 | loss scale: 1024.0 | grad norm: 27994.872 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
112502
+ time (ms)
112503
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.101563
112504
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.239360
112505
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.290825
112506
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.333122
112507
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.275614
112508
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.333908
112509
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.134140
112510
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.132914
112511
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.170101
112512
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.039771
112513
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.112406
112514
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.180832
112515
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:07.980029
112516
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.079841
112517
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:07.827544
112518
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.012831
112519
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:07.941577
112520
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.051269
112521
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.019638
112522
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.141076
112523
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.269351
112524
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.288065
112525
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.083403
112526
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.203051
112527
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.229694
112528
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.301282
112529
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.238132
112530
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:07.973572
112531
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.020849
112532
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.100762
112533
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.263260
112534
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.288527
112535
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:07.987462
112536
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:07.977094
112537
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.204458
112538
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.012745
112539
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.290414
112540
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.187495
112541
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.001168
112542
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:46:21 CEST)" was missed by 0:00:08.132191
112543
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.749619
112544
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.549838
112545
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.548609
112546
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.495488
112547
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.357278
112548
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.466977
112549
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.748844
112550
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.691313
112551
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.455484
112552
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.517284
112553
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.703740
112554
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.618747
112555
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.716998
112556
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.596535
112557
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.395754
112558
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.704189
112559
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.243230
112560
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.428566
112561
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.403123
112562
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.392766
112563
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.620106
112564
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.499089
112565
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.645409
112566
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.653803
112567
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.516474
112568
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.678967
112569
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.706096
112570
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.603149
112571
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.416793
112572
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.547792
112573
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.655654
112574
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.707143
112575
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.586415
112576
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.528711
112577
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.435949
112578
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.389842
112579
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.437077
112580
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.429012
112581
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.557420
112582
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:47:21 CEST)" was missed by 0:00:10.685654
112583
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.613708
112584
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.672035
112585
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.471028
112586
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.377884
112587
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.439700
112588
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.626136
112589
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.518930
112590
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.577507
112591
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.417929
112592
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.628975
112593
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.279686
112594
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.389368
112595
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.671274
112596
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.479229
112597
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.472275
112598
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.508262
112599
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.450548
112600
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.421508
112601
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.541163
112602
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.567819
112603
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.639412
112604
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.311687
112605
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.358955
112606
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.318160
112607
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.438910
112608
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.601372
112609
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.165680
112610
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.350973
112611
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.325582
112612
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.315206
112613
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.350839
112614
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.357818
112615
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.607512
112616
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.576275
112617
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.626655
112618
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.542618
112619
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.525630
112620
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.628564
112621
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.339306
112622
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:48:21 CEST)" was missed by 0:00:12.470287
112623
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.817252
112624
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.871588
112625
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.777234
112626
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.839036
112627
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.710960
112628
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.679021
112629
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.788724
112630
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.820851
112631
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.758227
112632
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.565018
112633
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.750316
112634
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.724859
112635
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.714510
112636
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.750126
112637
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.757142
112638
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.878566
112639
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.717509
112640
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.838234
112641
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.924915
112642
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.738588
112643
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 06:49:21 CEST)" was missed by 0:00:12.869558
112644
+ iteration 6570/ 159576 | consumed samples: 222400 | elapsed time per iteration (ms): 30604.9 | learning rate: 6.000E-05 | global batch size: 80 | lm loss: 6.300595E+00 | loss scale: 1024.0 | grad norm: 26978.212 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
112645
+ time (ms)