Hugging Face's logo

bigscience
/
tr8-104B-logs

TensorBoard
Model card Files Metrics Community
bigscience-bot commited on
Commit
d68e34f
•
1 Parent(s): ff31a34

new data

Browse files
Files changed (1) hide show
  1. logs/main_log.txt +238 -0
logs/main_log.txt CHANGED
@@ -115244,3 +115244,241 @@ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._mea
115244
  iteration 6790/ 159576 | consumed samples: 240752 | elapsed time per iteration (ms): 30867.9 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.349704E+00 | loss scale: 1024.0 | grad norm: 35833.809 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115245
  time (ms)
115246
  [2021-09-30 08:43:20] PULSE: tr8-104B is running for 4:51:13 since 2021-09-30T03:52:07 (1289770 on 'gpu_p13' partition (r6i4n[5-6,8],r6i5n[4-5],r7i0n[5-8],r7i1n0,r8i2n8,r8i4n1,r8i7n[3-8],r9i0n[0-8],r9i1n[0-8],r9i2n[3-8],r9i3n[7-8],r9i4n[0-2],r9i5n[2,5-7],r9i6n[2-8],r14i7n[1-6])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115244
  iteration 6790/ 159576 | consumed samples: 240752 | elapsed time per iteration (ms): 30867.9 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.349704E+00 | loss scale: 1024.0 | grad norm: 35833.809 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115245
  time (ms)
115246
  [2021-09-30 08:43:20] PULSE: tr8-104B is running for 4:51:13 since 2021-09-30T03:52:07 (1289770 on 'gpu_p13' partition (r6i4n[5-6,8],r6i5n[4-5],r7i0n[5-8],r7i1n0,r8i2n8,r8i4n1,r8i7n[3-8],r9i0n[0-8],r9i1n[0-8],r9i2n[3-8],r9i3n[7-8],r9i4n[0-2],r9i5n[2,5-7],r9i6n[2-8],r14i7n[1-6])
115247
+ iteration 6800/ 159576 | consumed samples: 241712 | elapsed time per iteration (ms): 30873.9 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.367813E+00 | loss scale: 1024.0 | grad norm: 41424.861 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115248
+ time (ms)
115249
+ iteration 6810/ 159576 | consumed samples: 242672 | elapsed time per iteration (ms): 30682.6 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.382545E+00 | loss scale: 1024.0 | grad norm: 34278.039 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115250
+ time (ms)
115251
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.136121
115252
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.054487
115253
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.291245
115254
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.086629
115255
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.024089
115256
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.291760
115257
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.278910
115258
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.104955
115259
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.241381
115260
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.184134
115261
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.242739
115262
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.083104
115263
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.336434
115264
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.293641
115265
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.337342
115266
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.115762
115267
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.023022
115268
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.137523
115269
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.206411
115270
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.294254
115271
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.016209
115272
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.043181
115273
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.272722
115274
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.304687
115275
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.266615
115276
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.016063
115277
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.233115
115278
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.173520
115279
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.207756
115280
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.190790
115281
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.144514
115282
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.104146
115283
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.135412
115284
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:57:21 CEST)" was missed by 0:00:03.004478
115285
+ iteration 6820/ 159576 | consumed samples: 243632 | elapsed time per iteration (ms): 30306.3 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.347304E+00 | loss scale: 1024.0 | grad norm: 43929.548 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115286
+ time (ms)
115287
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.709859
115288
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.628205
115289
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.864953
115290
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.660350
115291
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.852589
115292
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.757818
115293
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.597788
115294
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.865462
115295
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.910120
115296
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.816437
115297
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.689475
115298
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.815111
115299
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.656851
115300
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.840267
115301
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.867357
115302
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.678714
115303
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.867961
115304
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.711244
115305
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.846404
115306
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.589932
115307
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.589761
115308
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.596769
115309
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.718211
115310
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.911086
115311
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.616910
115312
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.780152
115313
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.878400
115314
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.747243
115315
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.764516
115316
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.806857
115317
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.781506
115318
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.677870
115319
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.518696
115320
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.404690
115321
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.709152
115322
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.564591
115323
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.578193
115324
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.550774
115325
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.554267
115326
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:58:21 CEST)" was missed by 0:00:03.557302
115327
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.861499
115328
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.016626
115329
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.779892
115330
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.004250
115331
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.812023
115332
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.909475
115333
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.968053
115334
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.017105
115335
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.841086
115336
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.830315
115337
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.966732
115338
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.702227
115339
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.749455
115340
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.556210
115341
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.716079
115342
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.670239
115343
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.061802
115344
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.808514
115345
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.991930
115346
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.705725
115347
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.741363
115348
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.748373
115349
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.018990
115350
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.062714
115351
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.768537
115352
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.998039
115353
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.931771
115354
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.030019
115355
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:06.019586
115356
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.741574
115357
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.862914
115358
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.898864
115359
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.933092
115360
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.916120
115361
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.958466
115362
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.869846
115363
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.708813
115364
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.829503
115365
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.860742
115366
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 08:59:21 CEST)" was missed by 0:00:05.729799
115367
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.885090
115368
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.803490
115369
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.040225
115370
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.027880
115371
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.864690
115372
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.835635
115373
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.933103
115374
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.773057
115375
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.832089
115376
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.040745
115377
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.693812
115378
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.085367
115379
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.991699
115380
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.015522
115381
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.579857
115382
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.771991
115383
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.990398
115384
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.725895
115385
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.043201
115386
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.739728
115387
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.086338
115388
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.729376
115389
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.765027
115390
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.792166
115391
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.854002
115392
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.021657
115393
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.765209
115394
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.042666
115395
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.893455
115396
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.886550
115397
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.922484
115398
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.955422
115399
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.982094
115400
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:08.053673
115401
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.732445
115402
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.956786
115403
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.939813
115404
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.853132
115405
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.884442
115406
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:00:21 CEST)" was missed by 0:00:07.753511
115407
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.001971
115408
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.157120
115409
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.920383
115410
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.049953
115411
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.157588
115412
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.202250
115413
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.144764
115414
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.981569
115415
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.952532
115416
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.696716
115417
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.108574
115418
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.842742
115419
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.889958
115420
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.948996
115421
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.132406
115422
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.856576
115423
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.810741
115424
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.159496
115425
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.203204
115426
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.003392
115427
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.039328
115428
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.909022
115429
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.970858
115430
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.072279
115431
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.160077
115432
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.882068
115433
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.888870
115434
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.138558
115435
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.170536
115436
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.107296
115437
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.846270
115438
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.881924
115439
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.098959
115440
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.010343
115441
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.073654
115442
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.849309
115443
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.969996
115444
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.056704
115445
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:10.001309
115446
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:01:21 CEST)" was missed by 0:00:09.870367
115447
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.858015
115448
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.776412
115449
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.905979
115450
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.808529
115451
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:13.000785
115452
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:13.013660
115453
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.837619
115454
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.745997
115455
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.805005
115456
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.988425
115457
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.552766
115458
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.712617
115459
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.666775
115460
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.826874
115461
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.964620
115462
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.698798
115463
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.737945
115464
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.744920
115465
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.765078
115466
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:13.016123
115467
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.738091
115468
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.702293
115469
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.859430
115470
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.895391
115471
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.994583
115472
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.928322
115473
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.705331
115474
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.866391
115475
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.955019
115476
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.826040
115477
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.912734
115478
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.929740
115479
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.857344
115480
+ WARNING:apscheduler.executors.default:Run time of job "BaseEmissionsTracker._measure_power (trigger: interval[0:01:00], next run at: 2021-09-30 09:02:21 CEST)" was missed by 0:00:12.726402
115481
+ iteration 6830/ 159576 | consumed samples: 244592 | elapsed time per iteration (ms): 31010.5 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.375375E+00 | loss scale: 1024.0 | grad norm: 40351.475 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115482
+ time (ms)
115483
+ iteration 6840/ 159576 | consumed samples: 245552 | elapsed time per iteration (ms): 30954.4 | learning rate: 6.000E-05 | global batch size: 96 | lm loss: 6.360943E+00 | loss scale: 1024.0 | grad norm: 42077.208 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
115484
+ time (ms)