[2024-09-05 08:53:10,744][00556] Saving configuration to /content/train_dir/default_experiment/config.json... [2024-09-05 08:53:10,748][00556] Rollout worker 0 uses device cpu [2024-09-05 08:53:10,751][00556] Rollout worker 1 uses device cpu [2024-09-05 08:53:10,754][00556] Rollout worker 2 uses device cpu [2024-09-05 08:53:10,758][00556] Rollout worker 3 uses device cpu [2024-09-05 08:53:10,764][00556] Rollout worker 4 uses device cpu [2024-09-05 08:53:10,768][00556] Rollout worker 5 uses device cpu [2024-09-05 08:53:10,770][00556] Rollout worker 6 uses device cpu [2024-09-05 08:53:10,773][00556] Rollout worker 7 uses device cpu [2024-09-05 08:53:11,017][00556] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 08:53:11,026][00556] InferenceWorker_p0-w0: min num requests: 2 [2024-09-05 08:53:11,074][00556] Starting all processes... [2024-09-05 08:53:11,079][00556] Starting process learner_proc0 [2024-09-05 08:53:12,271][00556] Starting all processes... [2024-09-05 08:53:12,347][00556] Starting process inference_proc0-0 [2024-09-05 08:53:12,348][00556] Starting process rollout_proc0 [2024-09-05 08:53:12,348][00556] Starting process rollout_proc1 [2024-09-05 08:53:12,353][00556] Starting process rollout_proc2 [2024-09-05 08:53:12,358][00556] Starting process rollout_proc3 [2024-09-05 08:53:12,358][00556] Starting process rollout_proc4 [2024-09-05 08:53:12,358][00556] Starting process rollout_proc5 [2024-09-05 08:53:12,358][00556] Starting process rollout_proc6 [2024-09-05 08:53:12,376][00556] Starting process rollout_proc7 [2024-09-05 08:53:29,018][02575] Worker 2 uses CPU cores [0] [2024-09-05 08:53:29,287][02573] Worker 0 uses CPU cores [0] [2024-09-05 08:53:29,529][02559] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 08:53:29,534][02559] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 [2024-09-05 08:53:29,589][02559] Num visible devices: 1 [2024-09-05 08:53:29,621][02559] Starting seed is not provided [2024-09-05 08:53:29,622][02559] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 08:53:29,623][02559] Initializing actor-critic model on device cuda:0 [2024-09-05 08:53:29,624][02559] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 08:53:29,627][02559] RunningMeanStd input shape: (1,) [2024-09-05 08:53:29,675][02578] Worker 6 uses CPU cores [0] [2024-09-05 08:53:29,725][02559] ConvEncoder: input_channels=3 [2024-09-05 08:53:29,748][02579] Worker 5 uses CPU cores [1] [2024-09-05 08:53:29,777][02577] Worker 4 uses CPU cores [0] [2024-09-05 08:53:29,849][02574] Worker 1 uses CPU cores [1] [2024-09-05 08:53:29,863][02572] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 08:53:29,864][02572] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 [2024-09-05 08:53:29,887][02572] Num visible devices: 1 [2024-09-05 08:53:29,912][02580] Worker 7 uses CPU cores [1] [2024-09-05 08:53:29,936][02576] Worker 3 uses CPU cores [1] [2024-09-05 08:53:30,040][02559] Conv encoder output size: 512 [2024-09-05 08:53:30,040][02559] Policy head output size: 512 [2024-09-05 08:53:30,099][02559] Created Actor Critic model with architecture: [2024-09-05 08:53:30,099][02559] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2024-09-05 08:53:30,405][02559] Using optimizer [2024-09-05 08:53:30,995][00556] Heartbeat connected on Batcher_0 [2024-09-05 08:53:31,018][00556] Heartbeat connected on InferenceWorker_p0-w0 [2024-09-05 08:53:31,036][00556] Heartbeat connected on RolloutWorker_w0 [2024-09-05 08:53:31,042][00556] Heartbeat connected on RolloutWorker_w1 [2024-09-05 08:53:31,046][00556] Heartbeat connected on RolloutWorker_w2 [2024-09-05 08:53:31,051][00556] Heartbeat connected on RolloutWorker_w3 [2024-09-05 08:53:31,056][00556] Heartbeat connected on RolloutWorker_w4 [2024-09-05 08:53:31,061][00556] Heartbeat connected on RolloutWorker_w5 [2024-09-05 08:53:31,073][00556] Heartbeat connected on RolloutWorker_w7 [2024-09-05 08:53:31,074][00556] Heartbeat connected on RolloutWorker_w6 [2024-09-05 08:53:31,117][02559] No checkpoints found [2024-09-05 08:53:31,117][02559] Did not load from checkpoint, starting from scratch! [2024-09-05 08:53:31,118][02559] Initialized policy 0 weights for model version 0 [2024-09-05 08:53:31,124][02559] LearnerWorker_p0 finished initialization! [2024-09-05 08:53:31,125][02559] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 08:53:31,125][00556] Heartbeat connected on LearnerWorker_p0 [2024-09-05 08:53:31,222][02572] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 08:53:31,223][02572] RunningMeanStd input shape: (1,) [2024-09-05 08:53:31,245][02572] ConvEncoder: input_channels=3 [2024-09-05 08:53:31,355][02572] Conv encoder output size: 512 [2024-09-05 08:53:31,355][02572] Policy head output size: 512 [2024-09-05 08:53:31,408][00556] Inference worker 0-0 is ready! [2024-09-05 08:53:31,409][00556] All inference workers are ready! Signal rollout workers to start! [2024-09-05 08:53:31,624][02579] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,626][02576] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,628][02580] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,631][02574] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,623][02577] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,628][02573] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,632][02578] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,643][02575] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 08:53:31,952][00556] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 08:53:33,104][02575] Decorrelating experience for 0 frames... [2024-09-05 08:53:33,105][02573] Decorrelating experience for 0 frames... [2024-09-05 08:53:33,107][02577] Decorrelating experience for 0 frames... [2024-09-05 08:53:33,112][02579] Decorrelating experience for 0 frames... [2024-09-05 08:53:33,114][02576] Decorrelating experience for 0 frames... [2024-09-05 08:53:33,116][02580] Decorrelating experience for 0 frames... [2024-09-05 08:53:34,507][02575] Decorrelating experience for 32 frames... [2024-09-05 08:53:34,516][02577] Decorrelating experience for 32 frames... [2024-09-05 08:53:34,526][02573] Decorrelating experience for 32 frames... [2024-09-05 08:53:35,033][02580] Decorrelating experience for 32 frames... [2024-09-05 08:53:35,038][02576] Decorrelating experience for 32 frames... [2024-09-05 08:53:35,049][02579] Decorrelating experience for 32 frames... [2024-09-05 08:53:36,952][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 08:53:37,369][02578] Decorrelating experience for 0 frames... [2024-09-05 08:53:37,756][02575] Decorrelating experience for 64 frames... [2024-09-05 08:53:37,766][02573] Decorrelating experience for 64 frames... [2024-09-05 08:53:37,783][02577] Decorrelating experience for 64 frames... [2024-09-05 08:53:37,917][02574] Decorrelating experience for 0 frames... [2024-09-05 08:53:38,322][02576] Decorrelating experience for 64 frames... [2024-09-05 08:53:38,324][02580] Decorrelating experience for 64 frames... [2024-09-05 08:53:38,324][02579] Decorrelating experience for 64 frames... [2024-09-05 08:53:38,956][02574] Decorrelating experience for 32 frames... [2024-09-05 08:53:39,862][02574] Decorrelating experience for 64 frames... [2024-09-05 08:53:40,106][02578] Decorrelating experience for 32 frames... [2024-09-05 08:53:40,288][02577] Decorrelating experience for 96 frames... [2024-09-05 08:53:40,432][02575] Decorrelating experience for 96 frames... [2024-09-05 08:53:40,460][02573] Decorrelating experience for 96 frames... [2024-09-05 08:53:41,303][02578] Decorrelating experience for 64 frames... [2024-09-05 08:53:41,724][02578] Decorrelating experience for 96 frames... [2024-09-05 08:53:41,880][02574] Decorrelating experience for 96 frames... [2024-09-05 08:53:41,952][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 08:53:42,060][02579] Decorrelating experience for 96 frames... [2024-09-05 08:53:42,221][02576] Decorrelating experience for 96 frames... [2024-09-05 08:53:42,574][02580] Decorrelating experience for 96 frames... [2024-09-05 08:53:46,009][02559] Signal inference workers to stop experience collection... [2024-09-05 08:53:46,038][02572] InferenceWorker_p0-w0: stopping experience collection [2024-09-05 08:53:46,952][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 156.9. Samples: 2354. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 08:53:46,955][00556] Avg episode reward: [(0, '1.943')] [2024-09-05 08:53:50,604][02559] Signal inference workers to resume experience collection... [2024-09-05 08:53:50,606][02572] InferenceWorker_p0-w0: resuming experience collection [2024-09-05 08:53:51,955][00556] Fps is (10 sec: 819.0, 60 sec: 409.5, 300 sec: 409.5). Total num frames: 8192. Throughput: 0: 117.7. Samples: 2354. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2024-09-05 08:53:51,958][00556] Avg episode reward: [(0, '2.574')] [2024-09-05 08:53:56,953][00556] Fps is (10 sec: 2047.7, 60 sec: 819.1, 300 sec: 819.1). Total num frames: 20480. Throughput: 0: 218.0. Samples: 5450. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:53:56,956][00556] Avg episode reward: [(0, '3.669')] [2024-09-05 08:54:01,476][02572] Updated weights for policy 0, policy_version 10 (0.0036) [2024-09-05 08:54:01,952][00556] Fps is (10 sec: 3277.7, 60 sec: 1365.3, 300 sec: 1365.3). Total num frames: 40960. Throughput: 0: 359.8. Samples: 10794. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:54:01,958][00556] Avg episode reward: [(0, '4.167')] [2024-09-05 08:54:06,952][00556] Fps is (10 sec: 4506.1, 60 sec: 1872.4, 300 sec: 1872.4). Total num frames: 65536. Throughput: 0: 404.2. Samples: 14148. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 08:54:06,955][00556] Avg episode reward: [(0, '4.348')] [2024-09-05 08:54:11,957][00556] Fps is (10 sec: 3684.4, 60 sec: 1945.3, 300 sec: 1945.3). Total num frames: 77824. Throughput: 0: 488.8. Samples: 19554. Policy #0 lag: (min: 0.0, avg: 0.2, max: 1.0) [2024-09-05 08:54:11,960][00556] Avg episode reward: [(0, '4.364')] [2024-09-05 08:54:12,410][02572] Updated weights for policy 0, policy_version 20 (0.0036) [2024-09-05 08:54:16,952][00556] Fps is (10 sec: 2867.3, 60 sec: 2093.5, 300 sec: 2093.5). Total num frames: 94208. Throughput: 0: 530.3. Samples: 23864. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 08:54:16,957][00556] Avg episode reward: [(0, '4.370')] [2024-09-05 08:54:21,952][00556] Fps is (10 sec: 3688.3, 60 sec: 2293.8, 300 sec: 2293.8). Total num frames: 114688. Throughput: 0: 605.3. Samples: 27238. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:54:21,959][00556] Avg episode reward: [(0, '4.396')] [2024-09-05 08:54:21,966][02559] Saving new best policy, reward=4.396! [2024-09-05 08:54:23,067][02572] Updated weights for policy 0, policy_version 30 (0.0028) [2024-09-05 08:54:26,952][00556] Fps is (10 sec: 4096.1, 60 sec: 2457.6, 300 sec: 2457.6). Total num frames: 135168. Throughput: 0: 754.7. Samples: 33962. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 08:54:26,956][00556] Avg episode reward: [(0, '4.516')] [2024-09-05 08:54:26,964][02559] Saving new best policy, reward=4.480! [2024-09-05 08:54:31,952][00556] Fps is (10 sec: 3686.4, 60 sec: 2525.9, 300 sec: 2525.9). Total num frames: 151552. Throughput: 0: 796.0. Samples: 38174. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:54:31,956][00556] Avg episode reward: [(0, '4.449')] [2024-09-05 08:54:35,372][02572] Updated weights for policy 0, policy_version 40 (0.0048) [2024-09-05 08:54:36,952][00556] Fps is (10 sec: 3276.8, 60 sec: 2798.9, 300 sec: 2583.6). Total num frames: 167936. Throughput: 0: 847.7. Samples: 40500. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:54:36,958][00556] Avg episode reward: [(0, '4.489')] [2024-09-05 08:54:36,960][02559] Saving new best policy, reward=4.489! [2024-09-05 08:54:41,952][00556] Fps is (10 sec: 3686.3, 60 sec: 3140.3, 300 sec: 2691.6). Total num frames: 188416. Throughput: 0: 918.5. Samples: 46780. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:54:41,955][00556] Avg episode reward: [(0, '4.491')] [2024-09-05 08:54:41,964][02559] Saving new best policy, reward=4.491! [2024-09-05 08:54:45,663][02572] Updated weights for policy 0, policy_version 50 (0.0039) [2024-09-05 08:54:46,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3413.3, 300 sec: 2730.7). Total num frames: 204800. Throughput: 0: 923.9. Samples: 52370. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:54:46,954][00556] Avg episode reward: [(0, '4.392')] [2024-09-05 08:54:51,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3550.0, 300 sec: 2764.8). Total num frames: 221184. Throughput: 0: 895.9. Samples: 54462. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 08:54:51,955][00556] Avg episode reward: [(0, '4.388')] [2024-09-05 08:54:56,952][00556] Fps is (10 sec: 3686.3, 60 sec: 3686.5, 300 sec: 2843.1). Total num frames: 241664. Throughput: 0: 908.2. Samples: 60420. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:54:56,955][00556] Avg episode reward: [(0, '4.398')] [2024-09-05 08:54:57,249][02572] Updated weights for policy 0, policy_version 60 (0.0023) [2024-09-05 08:55:01,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 2912.7). Total num frames: 262144. Throughput: 0: 948.9. Samples: 66564. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 08:55:01,958][00556] Avg episode reward: [(0, '4.391')] [2024-09-05 08:55:01,968][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000064_262144.pth... [2024-09-05 08:55:06,957][00556] Fps is (10 sec: 3684.5, 60 sec: 3549.6, 300 sec: 2931.7). Total num frames: 278528. Throughput: 0: 918.4. Samples: 68570. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:55:06,961][00556] Avg episode reward: [(0, '4.315')] [2024-09-05 08:55:09,404][02572] Updated weights for policy 0, policy_version 70 (0.0022) [2024-09-05 08:55:11,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.5, 300 sec: 2949.1). Total num frames: 294912. Throughput: 0: 875.2. Samples: 73346. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:55:11,956][00556] Avg episode reward: [(0, '4.475')] [2024-09-05 08:55:16,952][00556] Fps is (10 sec: 4098.3, 60 sec: 3754.7, 300 sec: 3042.7). Total num frames: 319488. Throughput: 0: 934.5. Samples: 80226. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:55:16,954][00556] Avg episode reward: [(0, '4.484')] [2024-09-05 08:55:18,464][02572] Updated weights for policy 0, policy_version 80 (0.0021) [2024-09-05 08:55:21,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3053.4). Total num frames: 335872. Throughput: 0: 956.0. Samples: 83518. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:55:21,957][00556] Avg episode reward: [(0, '4.532')] [2024-09-05 08:55:21,966][02559] Saving new best policy, reward=4.532! [2024-09-05 08:55:26,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.1, 300 sec: 3063.1). Total num frames: 352256. Throughput: 0: 906.7. Samples: 87580. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:55:26,959][00556] Avg episode reward: [(0, '4.366')] [2024-09-05 08:55:30,364][02572] Updated weights for policy 0, policy_version 90 (0.0023) [2024-09-05 08:55:31,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3686.4, 300 sec: 3106.1). Total num frames: 372736. Throughput: 0: 923.0. Samples: 93906. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:55:31,958][00556] Avg episode reward: [(0, '4.297')] [2024-09-05 08:55:36,954][00556] Fps is (10 sec: 4505.6, 60 sec: 3822.9, 300 sec: 3178.5). Total num frames: 397312. Throughput: 0: 951.7. Samples: 97290. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:55:36,958][00556] Avg episode reward: [(0, '4.458')] [2024-09-05 08:55:41,306][02572] Updated weights for policy 0, policy_version 100 (0.0022) [2024-09-05 08:55:41,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3686.4, 300 sec: 3150.8). Total num frames: 409600. Throughput: 0: 933.6. Samples: 102432. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:55:41,954][00556] Avg episode reward: [(0, '4.403')] [2024-09-05 08:55:46,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3185.8). Total num frames: 430080. Throughput: 0: 910.8. Samples: 107552. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 08:55:46,958][00556] Avg episode reward: [(0, '4.432')] [2024-09-05 08:55:51,956][00556] Fps is (10 sec: 3684.7, 60 sec: 3754.4, 300 sec: 3188.9). Total num frames: 446464. Throughput: 0: 942.2. Samples: 110968. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:55:51,960][00556] Avg episode reward: [(0, '4.526')] [2024-09-05 08:55:52,053][02572] Updated weights for policy 0, policy_version 110 (0.0031) [2024-09-05 08:55:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3192.1). Total num frames: 462848. Throughput: 0: 935.2. Samples: 115432. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:55:56,955][00556] Avg episode reward: [(0, '4.412')] [2024-09-05 08:56:01,957][00556] Fps is (10 sec: 2867.2, 60 sec: 3549.6, 300 sec: 3167.5). Total num frames: 475136. Throughput: 0: 862.4. Samples: 119040. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:56:01,959][00556] Avg episode reward: [(0, '4.470')] [2024-09-05 08:56:06,419][02572] Updated weights for policy 0, policy_version 120 (0.0060) [2024-09-05 08:56:06,952][00556] Fps is (10 sec: 2867.2, 60 sec: 3550.2, 300 sec: 3171.1). Total num frames: 491520. Throughput: 0: 839.8. Samples: 121310. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:56:06,959][00556] Avg episode reward: [(0, '4.585')] [2024-09-05 08:56:06,964][02559] Saving new best policy, reward=4.585! [2024-09-05 08:56:11,952][00556] Fps is (10 sec: 4097.8, 60 sec: 3686.4, 300 sec: 3225.6). Total num frames: 516096. Throughput: 0: 896.5. Samples: 127924. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 08:56:11,959][00556] Avg episode reward: [(0, '4.587')] [2024-09-05 08:56:11,968][02559] Saving new best policy, reward=4.587! [2024-09-05 08:56:16,540][02572] Updated weights for policy 0, policy_version 130 (0.0021) [2024-09-05 08:56:16,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3227.2). Total num frames: 532480. Throughput: 0: 882.6. Samples: 133622. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:56:16,954][00556] Avg episode reward: [(0, '4.587')] [2024-09-05 08:56:21,952][00556] Fps is (10 sec: 2867.3, 60 sec: 3481.6, 300 sec: 3204.5). Total num frames: 544768. Throughput: 0: 853.4. Samples: 135694. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:56:21,958][00556] Avg episode reward: [(0, '4.576')] [2024-09-05 08:56:26,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3618.1, 300 sec: 3253.4). Total num frames: 569344. Throughput: 0: 869.4. Samples: 141556. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:56:26,961][00556] Avg episode reward: [(0, '4.439')] [2024-09-05 08:56:27,656][02572] Updated weights for policy 0, policy_version 140 (0.0025) [2024-09-05 08:56:31,955][00556] Fps is (10 sec: 4504.0, 60 sec: 3617.9, 300 sec: 3276.7). Total num frames: 589824. Throughput: 0: 904.5. Samples: 148260. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:56:31,958][00556] Avg episode reward: [(0, '4.341')] [2024-09-05 08:56:36,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3254.7). Total num frames: 602112. Throughput: 0: 876.0. Samples: 150384. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:56:36,958][00556] Avg episode reward: [(0, '4.360')] [2024-09-05 08:56:39,747][02572] Updated weights for policy 0, policy_version 150 (0.0040) [2024-09-05 08:56:41,952][00556] Fps is (10 sec: 3277.9, 60 sec: 3549.9, 300 sec: 3276.8). Total num frames: 622592. Throughput: 0: 877.2. Samples: 154906. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:56:41,958][00556] Avg episode reward: [(0, '4.385')] [2024-09-05 08:56:46,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3297.8). Total num frames: 643072. Throughput: 0: 946.3. Samples: 161620. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:56:46,956][00556] Avg episode reward: [(0, '4.524')] [2024-09-05 08:56:48,843][02572] Updated weights for policy 0, policy_version 160 (0.0021) [2024-09-05 08:56:51,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3618.4, 300 sec: 3317.8). Total num frames: 663552. Throughput: 0: 968.2. Samples: 164880. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:56:51,955][00556] Avg episode reward: [(0, '4.675')] [2024-09-05 08:56:51,981][02559] Saving new best policy, reward=4.675! [2024-09-05 08:56:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3296.8). Total num frames: 675840. Throughput: 0: 910.7. Samples: 168904. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:56:56,955][00556] Avg episode reward: [(0, '4.592')] [2024-09-05 08:57:00,994][02572] Updated weights for policy 0, policy_version 170 (0.0019) [2024-09-05 08:57:01,952][00556] Fps is (10 sec: 3686.3, 60 sec: 3754.9, 300 sec: 3335.3). Total num frames: 700416. Throughput: 0: 922.6. Samples: 175140. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:57:01,955][00556] Avg episode reward: [(0, '4.742')] [2024-09-05 08:57:01,966][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000171_700416.pth... [2024-09-05 08:57:02,094][02559] Saving new best policy, reward=4.742! [2024-09-05 08:57:06,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3822.9, 300 sec: 3353.0). Total num frames: 720896. Throughput: 0: 947.5. Samples: 178330. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:57:06,963][00556] Avg episode reward: [(0, '4.563')] [2024-09-05 08:57:11,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3618.1, 300 sec: 3332.7). Total num frames: 733184. Throughput: 0: 931.1. Samples: 183454. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 08:57:11,959][00556] Avg episode reward: [(0, '4.692')] [2024-09-05 08:57:12,261][02572] Updated weights for policy 0, policy_version 180 (0.0021) [2024-09-05 08:57:16,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3349.6). Total num frames: 753664. Throughput: 0: 893.9. Samples: 188482. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 08:57:16,955][00556] Avg episode reward: [(0, '4.620')] [2024-09-05 08:57:21,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3365.8). Total num frames: 774144. Throughput: 0: 925.1. Samples: 192012. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:57:21,955][00556] Avg episode reward: [(0, '4.714')] [2024-09-05 08:57:22,052][02572] Updated weights for policy 0, policy_version 190 (0.0036) [2024-09-05 08:57:26,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3381.4). Total num frames: 794624. Throughput: 0: 970.7. Samples: 198588. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 08:57:26,959][00556] Avg episode reward: [(0, '4.770')] [2024-09-05 08:57:26,962][02559] Saving new best policy, reward=4.770! [2024-09-05 08:57:31,954][00556] Fps is (10 sec: 3276.1, 60 sec: 3618.2, 300 sec: 3362.1). Total num frames: 806912. Throughput: 0: 915.6. Samples: 202826. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 08:57:31,957][00556] Avg episode reward: [(0, '4.740')] [2024-09-05 08:57:33,993][02572] Updated weights for policy 0, policy_version 200 (0.0035) [2024-09-05 08:57:36,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3393.8). Total num frames: 831488. Throughput: 0: 913.2. Samples: 205974. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:57:36,954][00556] Avg episode reward: [(0, '4.396')] [2024-09-05 08:57:41,952][00556] Fps is (10 sec: 4916.0, 60 sec: 3891.2, 300 sec: 3424.3). Total num frames: 856064. Throughput: 0: 978.5. Samples: 212936. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:57:41,956][00556] Avg episode reward: [(0, '4.385')] [2024-09-05 08:57:43,251][02572] Updated weights for policy 0, policy_version 210 (0.0023) [2024-09-05 08:57:46,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3405.3). Total num frames: 868352. Throughput: 0: 946.0. Samples: 217710. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 08:57:46,954][00556] Avg episode reward: [(0, '4.545')] [2024-09-05 08:57:51,952][00556] Fps is (10 sec: 2867.3, 60 sec: 3686.4, 300 sec: 3402.8). Total num frames: 884736. Throughput: 0: 923.6. Samples: 219890. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:57:51,958][00556] Avg episode reward: [(0, '4.463')] [2024-09-05 08:57:54,725][02572] Updated weights for policy 0, policy_version 220 (0.0031) [2024-09-05 08:57:56,954][00556] Fps is (10 sec: 4095.1, 60 sec: 3891.1, 300 sec: 3431.3). Total num frames: 909312. Throughput: 0: 964.6. Samples: 226864. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 08:57:56,961][00556] Avg episode reward: [(0, '4.525')] [2024-09-05 08:58:01,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3822.9, 300 sec: 3443.7). Total num frames: 929792. Throughput: 0: 989.7. Samples: 233018. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:58:01,957][00556] Avg episode reward: [(0, '4.519')] [2024-09-05 08:58:05,896][02572] Updated weights for policy 0, policy_version 230 (0.0025) [2024-09-05 08:58:06,953][00556] Fps is (10 sec: 3277.1, 60 sec: 3686.3, 300 sec: 3425.7). Total num frames: 942080. Throughput: 0: 958.1. Samples: 235130. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:58:06,956][00556] Avg episode reward: [(0, '4.549')] [2024-09-05 08:58:11,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3452.3). Total num frames: 966656. Throughput: 0: 939.8. Samples: 240878. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:58:11,955][00556] Avg episode reward: [(0, '4.488')] [2024-09-05 08:58:15,295][02572] Updated weights for policy 0, policy_version 240 (0.0030) [2024-09-05 08:58:16,952][00556] Fps is (10 sec: 4915.7, 60 sec: 3959.4, 300 sec: 3478.0). Total num frames: 991232. Throughput: 0: 995.7. Samples: 247630. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:58:16,958][00556] Avg episode reward: [(0, '4.585')] [2024-09-05 08:58:21,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3460.4). Total num frames: 1003520. Throughput: 0: 980.7. Samples: 250104. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:58:21,954][00556] Avg episode reward: [(0, '4.596')] [2024-09-05 08:58:26,878][02572] Updated weights for policy 0, policy_version 250 (0.0026) [2024-09-05 08:58:26,952][00556] Fps is (10 sec: 3277.0, 60 sec: 3822.9, 300 sec: 3471.2). Total num frames: 1024000. Throughput: 0: 932.1. Samples: 254882. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:58:26,958][00556] Avg episode reward: [(0, '4.695')] [2024-09-05 08:58:31,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3959.6, 300 sec: 3540.6). Total num frames: 1044480. Throughput: 0: 983.9. Samples: 261984. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 08:58:31,959][00556] Avg episode reward: [(0, '4.680')] [2024-09-05 08:58:36,053][02572] Updated weights for policy 0, policy_version 260 (0.0023) [2024-09-05 08:58:36,952][00556] Fps is (10 sec: 4095.9, 60 sec: 3891.2, 300 sec: 3610.0). Total num frames: 1064960. Throughput: 0: 1014.8. Samples: 265554. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:58:36,955][00556] Avg episode reward: [(0, '4.601')] [2024-09-05 08:58:41,952][00556] Fps is (10 sec: 3686.5, 60 sec: 3754.7, 300 sec: 3665.6). Total num frames: 1081344. Throughput: 0: 954.7. Samples: 269822. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 08:58:41,954][00556] Avg episode reward: [(0, '4.539')] [2024-09-05 08:58:46,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3707.3). Total num frames: 1101824. Throughput: 0: 957.4. Samples: 276100. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:58:46,955][00556] Avg episode reward: [(0, '4.750')] [2024-09-05 08:58:47,228][02572] Updated weights for policy 0, policy_version 270 (0.0042) [2024-09-05 08:58:51,952][00556] Fps is (10 sec: 4505.5, 60 sec: 4027.7, 300 sec: 3748.9). Total num frames: 1126400. Throughput: 0: 987.6. Samples: 279572. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:58:51,955][00556] Avg episode reward: [(0, '5.012')] [2024-09-05 08:58:51,965][02559] Saving new best policy, reward=5.012! [2024-09-05 08:58:56,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.1, 300 sec: 3721.1). Total num frames: 1138688. Throughput: 0: 980.5. Samples: 285002. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:58:56,954][00556] Avg episode reward: [(0, '5.118')] [2024-09-05 08:58:57,014][02559] Saving new best policy, reward=5.118! [2024-09-05 08:58:58,601][02572] Updated weights for policy 0, policy_version 280 (0.0034) [2024-09-05 08:59:01,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3822.9, 300 sec: 3707.2). Total num frames: 1159168. Throughput: 0: 944.3. Samples: 290124. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:59:01,954][00556] Avg episode reward: [(0, '5.154')] [2024-09-05 08:59:01,964][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000283_1159168.pth... [2024-09-05 08:59:02,094][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000064_262144.pth [2024-09-05 08:59:02,110][02559] Saving new best policy, reward=5.154! [2024-09-05 08:59:06,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3959.6, 300 sec: 3735.1). Total num frames: 1179648. Throughput: 0: 961.7. Samples: 293382. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:59:06,955][00556] Avg episode reward: [(0, '5.208')] [2024-09-05 08:59:07,051][02559] Saving new best policy, reward=5.208! [2024-09-05 08:59:07,984][02572] Updated weights for policy 0, policy_version 290 (0.0031) [2024-09-05 08:59:11,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3748.9). Total num frames: 1200128. Throughput: 0: 1002.4. Samples: 299988. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 08:59:11,956][00556] Avg episode reward: [(0, '5.210')] [2024-09-05 08:59:11,975][02559] Saving new best policy, reward=5.210! [2024-09-05 08:59:16,957][00556] Fps is (10 sec: 3684.7, 60 sec: 3754.4, 300 sec: 3734.9). Total num frames: 1216512. Throughput: 0: 939.0. Samples: 304244. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:59:16,959][00556] Avg episode reward: [(0, '5.657')] [2024-09-05 08:59:16,962][02559] Saving new best policy, reward=5.657! [2024-09-05 08:59:19,836][02572] Updated weights for policy 0, policy_version 300 (0.0026) [2024-09-05 08:59:21,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3735.0). Total num frames: 1236992. Throughput: 0: 927.3. Samples: 307284. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 08:59:21,956][00556] Avg episode reward: [(0, '5.778')] [2024-09-05 08:59:21,966][02559] Saving new best policy, reward=5.778! [2024-09-05 08:59:26,952][00556] Fps is (10 sec: 4097.9, 60 sec: 3891.2, 300 sec: 3748.9). Total num frames: 1257472. Throughput: 0: 983.5. Samples: 314080. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:59:26,954][00556] Avg episode reward: [(0, '5.262')] [2024-09-05 08:59:29,677][02572] Updated weights for policy 0, policy_version 310 (0.0034) [2024-09-05 08:59:31,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3748.9). Total num frames: 1273856. Throughput: 0: 958.2. Samples: 319220. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:59:31,959][00556] Avg episode reward: [(0, '5.243')] [2024-09-05 08:59:36,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 1290240. Throughput: 0: 927.0. Samples: 321288. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:59:36,954][00556] Avg episode reward: [(0, '5.225')] [2024-09-05 08:59:40,707][02572] Updated weights for policy 0, policy_version 320 (0.0035) [2024-09-05 08:59:41,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 1314816. Throughput: 0: 955.8. Samples: 328014. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 08:59:41,959][00556] Avg episode reward: [(0, '5.492')] [2024-09-05 08:59:46,954][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 1335296. Throughput: 0: 976.9. Samples: 334086. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 08:59:46,957][00556] Avg episode reward: [(0, '5.472')] [2024-09-05 08:59:51,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 1347584. Throughput: 0: 950.0. Samples: 336132. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 08:59:51,959][00556] Avg episode reward: [(0, '5.338')] [2024-09-05 08:59:52,838][02572] Updated weights for policy 0, policy_version 330 (0.0021) [2024-09-05 08:59:56,952][00556] Fps is (10 sec: 2867.2, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 1363968. Throughput: 0: 903.8. Samples: 340660. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 08:59:56,959][00556] Avg episode reward: [(0, '5.482')] [2024-09-05 09:00:01,952][00556] Fps is (10 sec: 2867.2, 60 sec: 3618.1, 300 sec: 3721.2). Total num frames: 1376256. Throughput: 0: 904.1. Samples: 344924. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:00:01,960][00556] Avg episode reward: [(0, '5.405')] [2024-09-05 09:00:06,243][02572] Updated weights for policy 0, policy_version 340 (0.0029) [2024-09-05 09:00:06,952][00556] Fps is (10 sec: 2867.1, 60 sec: 3549.9, 300 sec: 3721.1). Total num frames: 1392640. Throughput: 0: 895.3. Samples: 347572. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:00:06,960][00556] Avg episode reward: [(0, '5.463')] [2024-09-05 09:00:11,952][00556] Fps is (10 sec: 3276.7, 60 sec: 3481.6, 300 sec: 3693.3). Total num frames: 1409024. Throughput: 0: 839.7. Samples: 351868. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:00:11,956][00556] Avg episode reward: [(0, '5.734')] [2024-09-05 09:00:16,692][02572] Updated weights for policy 0, policy_version 350 (0.0025) [2024-09-05 09:00:16,952][00556] Fps is (10 sec: 4096.1, 60 sec: 3618.4, 300 sec: 3721.1). Total num frames: 1433600. Throughput: 0: 876.7. Samples: 358670. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:00:16,957][00556] Avg episode reward: [(0, '5.920')] [2024-09-05 09:00:16,962][02559] Saving new best policy, reward=5.920! [2024-09-05 09:00:21,952][00556] Fps is (10 sec: 4505.7, 60 sec: 3618.1, 300 sec: 3735.0). Total num frames: 1454080. Throughput: 0: 906.8. Samples: 362094. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:00:21,955][00556] Avg episode reward: [(0, '5.709')] [2024-09-05 09:00:26,954][00556] Fps is (10 sec: 3276.0, 60 sec: 3481.5, 300 sec: 3707.2). Total num frames: 1466368. Throughput: 0: 858.9. Samples: 366668. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:00:26,957][00556] Avg episode reward: [(0, '5.779')] [2024-09-05 09:00:28,467][02572] Updated weights for policy 0, policy_version 360 (0.0023) [2024-09-05 09:00:31,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3693.3). Total num frames: 1486848. Throughput: 0: 857.0. Samples: 372652. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:00:31,954][00556] Avg episode reward: [(0, '6.184')] [2024-09-05 09:00:31,967][02559] Saving new best policy, reward=6.184! [2024-09-05 09:00:36,952][00556] Fps is (10 sec: 4506.7, 60 sec: 3686.4, 300 sec: 3735.0). Total num frames: 1511424. Throughput: 0: 885.4. Samples: 375976. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:00:36,957][00556] Avg episode reward: [(0, '6.284')] [2024-09-05 09:00:36,960][02559] Saving new best policy, reward=6.284! [2024-09-05 09:00:37,472][02572] Updated weights for policy 0, policy_version 370 (0.0038) [2024-09-05 09:00:41,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3721.1). Total num frames: 1527808. Throughput: 0: 910.5. Samples: 381632. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:00:41,958][00556] Avg episode reward: [(0, '6.069')] [2024-09-05 09:00:46,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3481.6, 300 sec: 3721.2). Total num frames: 1544192. Throughput: 0: 921.9. Samples: 386408. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:00:46,960][00556] Avg episode reward: [(0, '5.675')] [2024-09-05 09:00:49,353][02572] Updated weights for policy 0, policy_version 380 (0.0023) [2024-09-05 09:00:51,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3618.1, 300 sec: 3735.0). Total num frames: 1564672. Throughput: 0: 938.8. Samples: 389820. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:00:51,958][00556] Avg episode reward: [(0, '6.108')] [2024-09-05 09:00:56,957][00556] Fps is (10 sec: 4093.8, 60 sec: 3686.1, 300 sec: 3762.8). Total num frames: 1585152. Throughput: 0: 993.8. Samples: 396592. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:00:56,963][00556] Avg episode reward: [(0, '6.781')] [2024-09-05 09:00:56,972][02559] Saving new best policy, reward=6.781! [2024-09-05 09:01:00,208][02572] Updated weights for policy 0, policy_version 390 (0.0040) [2024-09-05 09:01:01,953][00556] Fps is (10 sec: 3685.8, 60 sec: 3754.6, 300 sec: 3762.7). Total num frames: 1601536. Throughput: 0: 933.3. Samples: 400670. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:01:01,956][00556] Avg episode reward: [(0, '6.826')] [2024-09-05 09:01:01,977][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000391_1601536.pth... [2024-09-05 09:01:02,141][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000171_700416.pth [2024-09-05 09:01:02,163][02559] Saving new best policy, reward=6.826! [2024-09-05 09:01:06,952][00556] Fps is (10 sec: 3688.4, 60 sec: 3822.9, 300 sec: 3748.9). Total num frames: 1622016. Throughput: 0: 917.7. Samples: 403390. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:01:06,957][00556] Avg episode reward: [(0, '7.158')] [2024-09-05 09:01:06,962][02559] Saving new best policy, reward=7.158! [2024-09-05 09:01:10,205][02572] Updated weights for policy 0, policy_version 400 (0.0035) [2024-09-05 09:01:11,952][00556] Fps is (10 sec: 4096.6, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 1642496. Throughput: 0: 968.6. Samples: 410254. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:01:11,957][00556] Avg episode reward: [(0, '7.466')] [2024-09-05 09:01:12,051][02559] Saving new best policy, reward=7.466! [2024-09-05 09:01:16,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 1658880. Throughput: 0: 951.7. Samples: 415478. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:01:16,959][00556] Avg episode reward: [(0, '7.832')] [2024-09-05 09:01:16,965][02559] Saving new best policy, reward=7.832! [2024-09-05 09:01:21,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 1675264. Throughput: 0: 925.4. Samples: 417618. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:01:21,956][00556] Avg episode reward: [(0, '7.690')] [2024-09-05 09:01:22,019][02572] Updated weights for policy 0, policy_version 410 (0.0043) [2024-09-05 09:01:26,952][00556] Fps is (10 sec: 4095.8, 60 sec: 3891.3, 300 sec: 3762.8). Total num frames: 1699840. Throughput: 0: 945.5. Samples: 424182. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:01:26,955][00556] Avg episode reward: [(0, '7.430')] [2024-09-05 09:01:31,101][02572] Updated weights for policy 0, policy_version 420 (0.0016) [2024-09-05 09:01:31,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3790.5). Total num frames: 1720320. Throughput: 0: 982.9. Samples: 430640. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:01:31,955][00556] Avg episode reward: [(0, '7.436')] [2024-09-05 09:01:36,952][00556] Fps is (10 sec: 3277.0, 60 sec: 3686.4, 300 sec: 3762.8). Total num frames: 1732608. Throughput: 0: 953.9. Samples: 432746. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:01:36,955][00556] Avg episode reward: [(0, '7.520')] [2024-09-05 09:01:41,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 1757184. Throughput: 0: 927.0. Samples: 438302. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:01:41,955][00556] Avg episode reward: [(0, '7.651')] [2024-09-05 09:01:42,579][02572] Updated weights for policy 0, policy_version 430 (0.0028) [2024-09-05 09:01:46,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 1777664. Throughput: 0: 988.5. Samples: 445150. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:01:46,954][00556] Avg episode reward: [(0, '7.644')] [2024-09-05 09:01:51,956][00556] Fps is (10 sec: 3684.9, 60 sec: 3822.7, 300 sec: 3790.5). Total num frames: 1794048. Throughput: 0: 987.6. Samples: 447834. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:01:51,965][00556] Avg episode reward: [(0, '8.112')] [2024-09-05 09:01:51,974][02559] Saving new best policy, reward=8.112! [2024-09-05 09:01:53,878][02572] Updated weights for policy 0, policy_version 440 (0.0063) [2024-09-05 09:01:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3755.0, 300 sec: 3762.8). Total num frames: 1810432. Throughput: 0: 936.1. Samples: 452378. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:01:56,954][00556] Avg episode reward: [(0, '8.305')] [2024-09-05 09:01:56,960][02559] Saving new best policy, reward=8.305! [2024-09-05 09:02:01,952][00556] Fps is (10 sec: 4097.7, 60 sec: 3891.3, 300 sec: 3776.7). Total num frames: 1835008. Throughput: 0: 975.0. Samples: 459352. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:01,954][00556] Avg episode reward: [(0, '9.176')] [2024-09-05 09:02:01,968][02559] Saving new best policy, reward=9.176! [2024-09-05 09:02:03,319][02572] Updated weights for policy 0, policy_version 450 (0.0032) [2024-09-05 09:02:06,955][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 1855488. Throughput: 0: 1003.6. Samples: 462780. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:06,962][00556] Avg episode reward: [(0, '10.076')] [2024-09-05 09:02:06,968][02559] Saving new best policy, reward=10.076! [2024-09-05 09:02:11,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 1867776. Throughput: 0: 953.9. Samples: 467108. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:11,954][00556] Avg episode reward: [(0, '10.471')] [2024-09-05 09:02:11,967][02559] Saving new best policy, reward=10.471! [2024-09-05 09:02:14,901][02572] Updated weights for policy 0, policy_version 460 (0.0047) [2024-09-05 09:02:16,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3790.5). Total num frames: 1892352. Throughput: 0: 946.5. Samples: 473234. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:02:16,955][00556] Avg episode reward: [(0, '10.517')] [2024-09-05 09:02:16,962][02559] Saving new best policy, reward=10.517! [2024-09-05 09:02:21,955][00556] Fps is (10 sec: 4504.0, 60 sec: 3959.2, 300 sec: 3790.5). Total num frames: 1912832. Throughput: 0: 974.9. Samples: 476620. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:21,958][00556] Avg episode reward: [(0, '11.655')] [2024-09-05 09:02:21,973][02559] Saving new best policy, reward=11.655! [2024-09-05 09:02:25,339][02572] Updated weights for policy 0, policy_version 470 (0.0032) [2024-09-05 09:02:26,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.0, 300 sec: 3804.4). Total num frames: 1929216. Throughput: 0: 969.5. Samples: 481928. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:02:26,954][00556] Avg episode reward: [(0, '12.074')] [2024-09-05 09:02:26,959][02559] Saving new best policy, reward=12.074! [2024-09-05 09:02:31,952][00556] Fps is (10 sec: 3277.9, 60 sec: 3754.7, 300 sec: 3776.6). Total num frames: 1945600. Throughput: 0: 934.3. Samples: 487196. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:31,955][00556] Avg episode reward: [(0, '13.274')] [2024-09-05 09:02:31,967][02559] Saving new best policy, reward=13.274! [2024-09-05 09:02:35,708][02572] Updated weights for policy 0, policy_version 480 (0.0036) [2024-09-05 09:02:36,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3959.5, 300 sec: 3776.7). Total num frames: 1970176. Throughput: 0: 951.1. Samples: 490630. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:36,954][00556] Avg episode reward: [(0, '13.427')] [2024-09-05 09:02:36,957][02559] Saving new best policy, reward=13.427! [2024-09-05 09:02:41,952][00556] Fps is (10 sec: 4096.1, 60 sec: 3822.9, 300 sec: 3790.5). Total num frames: 1986560. Throughput: 0: 986.8. Samples: 496784. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:41,957][00556] Avg episode reward: [(0, '13.273')] [2024-09-05 09:02:46,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3790.5). Total num frames: 2002944. Throughput: 0: 926.9. Samples: 501062. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:02:46,954][00556] Avg episode reward: [(0, '13.703')] [2024-09-05 09:02:46,960][02559] Saving new best policy, reward=13.703! [2024-09-05 09:02:47,654][02572] Updated weights for policy 0, policy_version 490 (0.0031) [2024-09-05 09:02:51,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.2, 300 sec: 3776.7). Total num frames: 2023424. Throughput: 0: 924.5. Samples: 504384. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:02:51,954][00556] Avg episode reward: [(0, '12.837')] [2024-09-05 09:02:56,554][02572] Updated weights for policy 0, policy_version 500 (0.0014) [2024-09-05 09:02:56,952][00556] Fps is (10 sec: 4505.5, 60 sec: 3959.5, 300 sec: 3790.5). Total num frames: 2048000. Throughput: 0: 985.4. Samples: 511452. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:02:56,954][00556] Avg episode reward: [(0, '13.359')] [2024-09-05 09:03:01,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3790.6). Total num frames: 2060288. Throughput: 0: 950.6. Samples: 516012. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:01,954][00556] Avg episode reward: [(0, '13.143')] [2024-09-05 09:03:01,967][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000503_2060288.pth... [2024-09-05 09:03:02,139][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000283_1159168.pth [2024-09-05 09:03:06,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 2080768. Throughput: 0: 935.3. Samples: 518704. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:03:06,963][00556] Avg episode reward: [(0, '13.856')] [2024-09-05 09:03:06,972][02559] Saving new best policy, reward=13.856! [2024-09-05 09:03:07,990][02572] Updated weights for policy 0, policy_version 510 (0.0042) [2024-09-05 09:03:11,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3776.7). Total num frames: 2105344. Throughput: 0: 970.0. Samples: 525578. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:03:11,955][00556] Avg episode reward: [(0, '14.154')] [2024-09-05 09:03:11,967][02559] Saving new best policy, reward=14.154! [2024-09-05 09:03:16,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3790.5). Total num frames: 2121728. Throughput: 0: 973.4. Samples: 531000. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:16,960][00556] Avg episode reward: [(0, '15.343')] [2024-09-05 09:03:16,961][02559] Saving new best policy, reward=15.343! [2024-09-05 09:03:19,420][02572] Updated weights for policy 0, policy_version 520 (0.0025) [2024-09-05 09:03:21,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.9, 300 sec: 3776.6). Total num frames: 2138112. Throughput: 0: 944.2. Samples: 533118. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:03:21,961][00556] Avg episode reward: [(0, '15.865')] [2024-09-05 09:03:21,969][02559] Saving new best policy, reward=15.865! [2024-09-05 09:03:26,952][00556] Fps is (10 sec: 4095.9, 60 sec: 3891.2, 300 sec: 3790.5). Total num frames: 2162688. Throughput: 0: 950.7. Samples: 539566. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:03:26,961][00556] Avg episode reward: [(0, '16.982')] [2024-09-05 09:03:26,964][02559] Saving new best policy, reward=16.982! [2024-09-05 09:03:28,607][02572] Updated weights for policy 0, policy_version 530 (0.0034) [2024-09-05 09:03:31,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3790.5). Total num frames: 2183168. Throughput: 0: 1002.1. Samples: 546156. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:31,960][00556] Avg episode reward: [(0, '15.947')] [2024-09-05 09:03:36,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3754.7, 300 sec: 3776.6). Total num frames: 2195456. Throughput: 0: 975.4. Samples: 548278. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:36,958][00556] Avg episode reward: [(0, '14.973')] [2024-09-05 09:03:40,112][02572] Updated weights for policy 0, policy_version 540 (0.0031) [2024-09-05 09:03:41,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 2215936. Throughput: 0: 941.7. Samples: 553830. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:41,954][00556] Avg episode reward: [(0, '14.413')] [2024-09-05 09:03:46,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3776.7). Total num frames: 2240512. Throughput: 0: 995.5. Samples: 560810. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:46,956][00556] Avg episode reward: [(0, '14.327')] [2024-09-05 09:03:49,916][02572] Updated weights for policy 0, policy_version 550 (0.0037) [2024-09-05 09:03:51,952][00556] Fps is (10 sec: 4095.8, 60 sec: 3891.2, 300 sec: 3790.5). Total num frames: 2256896. Throughput: 0: 994.9. Samples: 563476. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:03:51,959][00556] Avg episode reward: [(0, '15.265')] [2024-09-05 09:03:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 2273280. Throughput: 0: 943.1. Samples: 568018. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:03:56,955][00556] Avg episode reward: [(0, '16.985')] [2024-09-05 09:03:56,957][02559] Saving new best policy, reward=16.985! [2024-09-05 09:04:01,955][00556] Fps is (10 sec: 3275.9, 60 sec: 3822.7, 300 sec: 3762.7). Total num frames: 2289664. Throughput: 0: 939.7. Samples: 573290. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:04:01,957][00556] Avg episode reward: [(0, '16.565')] [2024-09-05 09:04:02,489][02572] Updated weights for policy 0, policy_version 560 (0.0042) [2024-09-05 09:04:06,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 2306048. Throughput: 0: 940.0. Samples: 575416. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:04:06,954][00556] Avg episode reward: [(0, '16.400')] [2024-09-05 09:04:11,952][00556] Fps is (10 sec: 2868.0, 60 sec: 3549.9, 300 sec: 3735.1). Total num frames: 2318336. Throughput: 0: 888.5. Samples: 579548. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:04:11,955][00556] Avg episode reward: [(0, '15.898')] [2024-09-05 09:04:14,977][02572] Updated weights for policy 0, policy_version 570 (0.0029) [2024-09-05 09:04:16,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 2342912. Throughput: 0: 881.2. Samples: 585810. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:04:16,954][00556] Avg episode reward: [(0, '14.277')] [2024-09-05 09:04:21,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 2363392. Throughput: 0: 909.5. Samples: 589206. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:04:21,954][00556] Avg episode reward: [(0, '14.302')] [2024-09-05 09:04:24,997][02572] Updated weights for policy 0, policy_version 580 (0.0059) [2024-09-05 09:04:26,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3618.2, 300 sec: 3748.9). Total num frames: 2379776. Throughput: 0: 908.1. Samples: 594696. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:04:26,959][00556] Avg episode reward: [(0, '15.926')] [2024-09-05 09:04:31,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3748.9). Total num frames: 2396160. Throughput: 0: 870.8. Samples: 599994. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:04:31,956][00556] Avg episode reward: [(0, '17.115')] [2024-09-05 09:04:31,977][02559] Saving new best policy, reward=17.115! [2024-09-05 09:04:35,527][02572] Updated weights for policy 0, policy_version 590 (0.0030) [2024-09-05 09:04:36,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 2420736. Throughput: 0: 886.4. Samples: 603364. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:04:36,957][00556] Avg episode reward: [(0, '18.489')] [2024-09-05 09:04:36,961][02559] Saving new best policy, reward=18.489! [2024-09-05 09:04:41,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3735.0). Total num frames: 2437120. Throughput: 0: 927.1. Samples: 609738. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:04:41,954][00556] Avg episode reward: [(0, '19.600')] [2024-09-05 09:04:41,969][02559] Saving new best policy, reward=19.600! [2024-09-05 09:04:46,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3748.9). Total num frames: 2453504. Throughput: 0: 904.6. Samples: 613996. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:04:46,957][00556] Avg episode reward: [(0, '19.410')] [2024-09-05 09:04:47,461][02572] Updated weights for policy 0, policy_version 600 (0.0025) [2024-09-05 09:04:51,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3776.6). Total num frames: 2478080. Throughput: 0: 932.3. Samples: 617370. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:04:51,955][00556] Avg episode reward: [(0, '18.704')] [2024-09-05 09:04:56,122][02572] Updated weights for policy 0, policy_version 610 (0.0029) [2024-09-05 09:04:56,954][00556] Fps is (10 sec: 4504.5, 60 sec: 3754.5, 300 sec: 3804.4). Total num frames: 2498560. Throughput: 0: 996.5. Samples: 624394. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:04:56,957][00556] Avg episode reward: [(0, '18.206')] [2024-09-05 09:05:01,959][00556] Fps is (10 sec: 3683.8, 60 sec: 3754.4, 300 sec: 3804.3). Total num frames: 2514944. Throughput: 0: 961.0. Samples: 629062. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:05:01,966][00556] Avg episode reward: [(0, '19.347')] [2024-09-05 09:05:01,979][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000614_2514944.pth... [2024-09-05 09:05:02,165][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000391_1601536.pth [2024-09-05 09:05:06,952][00556] Fps is (10 sec: 3687.3, 60 sec: 3822.9, 300 sec: 3818.3). Total num frames: 2535424. Throughput: 0: 942.5. Samples: 631618. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:05:06,957][00556] Avg episode reward: [(0, '18.625')] [2024-09-05 09:05:07,616][02572] Updated weights for policy 0, policy_version 620 (0.0048) [2024-09-05 09:05:11,952][00556] Fps is (10 sec: 4099.0, 60 sec: 3959.5, 300 sec: 3804.4). Total num frames: 2555904. Throughput: 0: 977.4. Samples: 638680. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:05:11,957][00556] Avg episode reward: [(0, '18.854')] [2024-09-05 09:05:16,952][00556] Fps is (10 sec: 4096.1, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 2576384. Throughput: 0: 982.8. Samples: 644218. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:05:16,958][00556] Avg episode reward: [(0, '18.050')] [2024-09-05 09:05:18,526][02572] Updated weights for policy 0, policy_version 630 (0.0013) [2024-09-05 09:05:21,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3818.3). Total num frames: 2592768. Throughput: 0: 952.6. Samples: 646230. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:05:21,957][00556] Avg episode reward: [(0, '17.419')] [2024-09-05 09:05:26,952][00556] Fps is (10 sec: 3686.3, 60 sec: 3891.2, 300 sec: 3818.3). Total num frames: 2613248. Throughput: 0: 951.6. Samples: 652558. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:05:26,958][00556] Avg episode reward: [(0, '17.627')] [2024-09-05 09:05:28,272][02572] Updated weights for policy 0, policy_version 640 (0.0025) [2024-09-05 09:05:31,952][00556] Fps is (10 sec: 4095.8, 60 sec: 3959.4, 300 sec: 3804.4). Total num frames: 2633728. Throughput: 0: 1000.1. Samples: 659002. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:05:31,956][00556] Avg episode reward: [(0, '16.934')] [2024-09-05 09:05:36,954][00556] Fps is (10 sec: 3276.1, 60 sec: 3754.5, 300 sec: 3790.5). Total num frames: 2646016. Throughput: 0: 971.5. Samples: 661090. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:05:36,959][00556] Avg episode reward: [(0, '16.916')] [2024-09-05 09:05:40,083][02572] Updated weights for policy 0, policy_version 650 (0.0059) [2024-09-05 09:05:41,952][00556] Fps is (10 sec: 3686.6, 60 sec: 3891.2, 300 sec: 3818.3). Total num frames: 2670592. Throughput: 0: 936.6. Samples: 666540. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:05:41,954][00556] Avg episode reward: [(0, '17.948')] [2024-09-05 09:05:46,952][00556] Fps is (10 sec: 4506.6, 60 sec: 3959.5, 300 sec: 3818.3). Total num frames: 2691072. Throughput: 0: 987.7. Samples: 673502. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:05:46,959][00556] Avg episode reward: [(0, '18.330')] [2024-09-05 09:05:49,757][02572] Updated weights for policy 0, policy_version 660 (0.0029) [2024-09-05 09:05:51,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3804.5). Total num frames: 2707456. Throughput: 0: 987.9. Samples: 676074. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:05:51,955][00556] Avg episode reward: [(0, '18.467')] [2024-09-05 09:05:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.8, 300 sec: 3804.4). Total num frames: 2723840. Throughput: 0: 927.3. Samples: 680408. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:05:56,959][00556] Avg episode reward: [(0, '19.145')] [2024-09-05 09:06:00,976][02572] Updated weights for policy 0, policy_version 670 (0.0020) [2024-09-05 09:06:01,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.7, 300 sec: 3818.3). Total num frames: 2748416. Throughput: 0: 954.3. Samples: 687160. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:01,959][00556] Avg episode reward: [(0, '18.104')] [2024-09-05 09:06:06,958][00556] Fps is (10 sec: 4093.6, 60 sec: 3822.6, 300 sec: 3804.3). Total num frames: 2764800. Throughput: 0: 987.0. Samples: 690650. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:06,960][00556] Avg episode reward: [(0, '17.727')] [2024-09-05 09:06:11,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3804.4). Total num frames: 2781184. Throughput: 0: 946.1. Samples: 695132. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:11,959][00556] Avg episode reward: [(0, '18.652')] [2024-09-05 09:06:12,860][02572] Updated weights for policy 0, policy_version 680 (0.0020) [2024-09-05 09:06:16,952][00556] Fps is (10 sec: 3688.5, 60 sec: 3754.7, 300 sec: 3818.3). Total num frames: 2801664. Throughput: 0: 935.4. Samples: 701096. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:16,959][00556] Avg episode reward: [(0, '18.219')] [2024-09-05 09:06:21,936][02572] Updated weights for policy 0, policy_version 690 (0.0024) [2024-09-05 09:06:21,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3818.3). Total num frames: 2826240. Throughput: 0: 960.4. Samples: 704304. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:21,959][00556] Avg episode reward: [(0, '17.381')] [2024-09-05 09:06:26,957][00556] Fps is (10 sec: 3684.4, 60 sec: 3754.3, 300 sec: 3790.5). Total num frames: 2838528. Throughput: 0: 964.2. Samples: 709936. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:26,968][00556] Avg episode reward: [(0, '18.590')] [2024-09-05 09:06:31,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3818.3). Total num frames: 2859008. Throughput: 0: 919.0. Samples: 714858. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:06:31,955][00556] Avg episode reward: [(0, '18.294')] [2024-09-05 09:06:33,528][02572] Updated weights for policy 0, policy_version 700 (0.0030) [2024-09-05 09:06:36,952][00556] Fps is (10 sec: 4098.3, 60 sec: 3891.3, 300 sec: 3804.4). Total num frames: 2879488. Throughput: 0: 939.3. Samples: 718344. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:06:36,955][00556] Avg episode reward: [(0, '18.017')] [2024-09-05 09:06:41,952][00556] Fps is (10 sec: 4095.8, 60 sec: 3822.9, 300 sec: 3804.4). Total num frames: 2899968. Throughput: 0: 992.9. Samples: 725090. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:06:41,956][00556] Avg episode reward: [(0, '19.009')] [2024-09-05 09:06:44,148][02572] Updated weights for policy 0, policy_version 710 (0.0023) [2024-09-05 09:06:46,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3790.6). Total num frames: 2912256. Throughput: 0: 936.5. Samples: 729302. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:46,954][00556] Avg episode reward: [(0, '19.771')] [2024-09-05 09:06:46,961][02559] Saving new best policy, reward=19.771! [2024-09-05 09:06:51,952][00556] Fps is (10 sec: 3686.6, 60 sec: 3822.9, 300 sec: 3818.3). Total num frames: 2936832. Throughput: 0: 925.4. Samples: 732288. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:06:51,958][00556] Avg episode reward: [(0, '21.126')] [2024-09-05 09:06:51,969][02559] Saving new best policy, reward=21.126! [2024-09-05 09:06:54,447][02572] Updated weights for policy 0, policy_version 720 (0.0018) [2024-09-05 09:06:56,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 2957312. Throughput: 0: 973.4. Samples: 738936. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:06:56,959][00556] Avg episode reward: [(0, '21.435')] [2024-09-05 09:06:56,962][02559] Saving new best policy, reward=21.435! [2024-09-05 09:07:01,953][00556] Fps is (10 sec: 3685.9, 60 sec: 3754.6, 300 sec: 3790.5). Total num frames: 2973696. Throughput: 0: 949.7. Samples: 743834. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:07:01,958][00556] Avg episode reward: [(0, '20.289')] [2024-09-05 09:07:01,972][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000726_2973696.pth... [2024-09-05 09:07:02,165][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000503_2060288.pth [2024-09-05 09:07:06,243][02572] Updated weights for policy 0, policy_version 730 (0.0016) [2024-09-05 09:07:06,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3755.0, 300 sec: 3804.4). Total num frames: 2990080. Throughput: 0: 925.8. Samples: 745966. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:07:06,955][00556] Avg episode reward: [(0, '19.439')] [2024-09-05 09:07:11,952][00556] Fps is (10 sec: 4096.5, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 3014656. Throughput: 0: 954.9. Samples: 752902. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:07:11,958][00556] Avg episode reward: [(0, '20.353')] [2024-09-05 09:07:15,630][02572] Updated weights for policy 0, policy_version 740 (0.0027) [2024-09-05 09:07:16,952][00556] Fps is (10 sec: 4095.8, 60 sec: 3822.9, 300 sec: 3790.6). Total num frames: 3031040. Throughput: 0: 977.7. Samples: 758856. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:07:16,960][00556] Avg episode reward: [(0, '19.309')] [2024-09-05 09:07:21,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3790.5). Total num frames: 3047424. Throughput: 0: 945.6. Samples: 760896. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:07:21,958][00556] Avg episode reward: [(0, '20.002')] [2024-09-05 09:07:26,737][02572] Updated weights for policy 0, policy_version 750 (0.0017) [2024-09-05 09:07:26,952][00556] Fps is (10 sec: 4096.2, 60 sec: 3891.6, 300 sec: 3818.3). Total num frames: 3072000. Throughput: 0: 930.9. Samples: 766982. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:07:26,956][00556] Avg episode reward: [(0, '20.515')] [2024-09-05 09:07:31,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 3092480. Throughput: 0: 992.8. Samples: 773980. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:07:31,956][00556] Avg episode reward: [(0, '21.027')] [2024-09-05 09:07:36,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3804.4). Total num frames: 3108864. Throughput: 0: 972.6. Samples: 776056. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:07:36,955][00556] Avg episode reward: [(0, '21.848')] [2024-09-05 09:07:36,962][02559] Saving new best policy, reward=21.848! [2024-09-05 09:07:38,387][02572] Updated weights for policy 0, policy_version 760 (0.0040) [2024-09-05 09:07:41,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3804.4). Total num frames: 3125248. Throughput: 0: 936.3. Samples: 781070. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:07:41,958][00556] Avg episode reward: [(0, '22.179')] [2024-09-05 09:07:42,007][02559] Saving new best policy, reward=22.179! [2024-09-05 09:07:46,952][00556] Fps is (10 sec: 4095.9, 60 sec: 3959.5, 300 sec: 3818.3). Total num frames: 3149824. Throughput: 0: 979.6. Samples: 787914. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:07:46,956][00556] Avg episode reward: [(0, '22.185')] [2024-09-05 09:07:46,959][02559] Saving new best policy, reward=22.185! [2024-09-05 09:07:47,535][02572] Updated weights for policy 0, policy_version 770 (0.0035) [2024-09-05 09:07:51,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3790.5). Total num frames: 3166208. Throughput: 0: 995.9. Samples: 790780. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:07:51,954][00556] Avg episode reward: [(0, '22.584')] [2024-09-05 09:07:51,974][02559] Saving new best policy, reward=22.584! [2024-09-05 09:07:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3804.4). Total num frames: 3182592. Throughput: 0: 933.0. Samples: 794888. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:07:56,957][00556] Avg episode reward: [(0, '23.748')] [2024-09-05 09:07:56,963][02559] Saving new best policy, reward=23.748! [2024-09-05 09:07:59,492][02572] Updated weights for policy 0, policy_version 780 (0.0047) [2024-09-05 09:08:01,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.0, 300 sec: 3804.4). Total num frames: 3203072. Throughput: 0: 946.5. Samples: 801446. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:08:01,958][00556] Avg episode reward: [(0, '23.655')] [2024-09-05 09:08:06,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 3215360. Throughput: 0: 945.8. Samples: 803458. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:08:06,961][00556] Avg episode reward: [(0, '23.804')] [2024-09-05 09:08:06,963][02559] Saving new best policy, reward=23.804! [2024-09-05 09:08:11,956][00556] Fps is (10 sec: 2456.5, 60 sec: 3549.6, 300 sec: 3748.8). Total num frames: 3227648. Throughput: 0: 889.5. Samples: 807012. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:08:11,959][00556] Avg episode reward: [(0, '23.020')] [2024-09-05 09:08:13,993][02572] Updated weights for policy 0, policy_version 790 (0.0023) [2024-09-05 09:08:16,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.2, 300 sec: 3762.8). Total num frames: 3248128. Throughput: 0: 853.1. Samples: 812370. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:08:16,957][00556] Avg episode reward: [(0, '23.317')] [2024-09-05 09:08:21,952][00556] Fps is (10 sec: 4097.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 3268608. Throughput: 0: 883.7. Samples: 815824. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:08:21,961][00556] Avg episode reward: [(0, '20.902')] [2024-09-05 09:08:22,919][02572] Updated weights for policy 0, policy_version 800 (0.0040) [2024-09-05 09:08:26,956][00556] Fps is (10 sec: 4094.2, 60 sec: 3617.9, 300 sec: 3748.8). Total num frames: 3289088. Throughput: 0: 914.2. Samples: 822212. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:08:26,958][00556] Avg episode reward: [(0, '19.727')] [2024-09-05 09:08:31,952][00556] Fps is (10 sec: 3686.3, 60 sec: 3549.8, 300 sec: 3762.8). Total num frames: 3305472. Throughput: 0: 856.4. Samples: 826450. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:08:31,955][00556] Avg episode reward: [(0, '18.975')] [2024-09-05 09:08:34,614][02572] Updated weights for policy 0, policy_version 810 (0.0056) [2024-09-05 09:08:36,952][00556] Fps is (10 sec: 3688.0, 60 sec: 3618.1, 300 sec: 3762.8). Total num frames: 3325952. Throughput: 0: 870.2. Samples: 829938. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:08:36,960][00556] Avg episode reward: [(0, '19.685')] [2024-09-05 09:08:41,952][00556] Fps is (10 sec: 4505.7, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 3350528. Throughput: 0: 933.8. Samples: 836908. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:08:41,959][00556] Avg episode reward: [(0, '19.820')] [2024-09-05 09:08:44,610][02572] Updated weights for policy 0, policy_version 820 (0.0023) [2024-09-05 09:08:46,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3549.9, 300 sec: 3748.9). Total num frames: 3362816. Throughput: 0: 892.7. Samples: 841618. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:08:46,955][00556] Avg episode reward: [(0, '19.537')] [2024-09-05 09:08:51,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.1, 300 sec: 3762.8). Total num frames: 3383296. Throughput: 0: 907.2. Samples: 844282. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:08:51,960][00556] Avg episode reward: [(0, '20.711')] [2024-09-05 09:08:54,929][02572] Updated weights for policy 0, policy_version 830 (0.0025) [2024-09-05 09:08:56,952][00556] Fps is (10 sec: 4505.7, 60 sec: 3754.7, 300 sec: 3790.6). Total num frames: 3407872. Throughput: 0: 980.1. Samples: 851110. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:08:56,960][00556] Avg episode reward: [(0, '20.926')] [2024-09-05 09:09:01,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3790.5). Total num frames: 3424256. Throughput: 0: 984.7. Samples: 856680. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:09:01,958][00556] Avg episode reward: [(0, '19.977')] [2024-09-05 09:09:01,968][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000836_3424256.pth... [2024-09-05 09:09:02,109][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000614_2514944.pth [2024-09-05 09:09:06,616][02572] Updated weights for policy 0, policy_version 840 (0.0041) [2024-09-05 09:09:06,956][00556] Fps is (10 sec: 3275.3, 60 sec: 3754.4, 300 sec: 3804.4). Total num frames: 3440640. Throughput: 0: 953.8. Samples: 858748. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:09:06,959][00556] Avg episode reward: [(0, '20.138')] [2024-09-05 09:09:11,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3959.8, 300 sec: 3804.4). Total num frames: 3465216. Throughput: 0: 955.1. Samples: 865188. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:09:11,959][00556] Avg episode reward: [(0, '20.356')] [2024-09-05 09:09:15,463][02572] Updated weights for policy 0, policy_version 850 (0.0034) [2024-09-05 09:09:16,952][00556] Fps is (10 sec: 4507.5, 60 sec: 3959.5, 300 sec: 3804.4). Total num frames: 3485696. Throughput: 0: 1006.8. Samples: 871754. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:09:16,958][00556] Avg episode reward: [(0, '21.021')] [2024-09-05 09:09:21,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3822.9, 300 sec: 3790.5). Total num frames: 3497984. Throughput: 0: 975.6. Samples: 873842. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:09:21,958][00556] Avg episode reward: [(0, '20.843')] [2024-09-05 09:09:26,952][00556] Fps is (10 sec: 3276.7, 60 sec: 3823.2, 300 sec: 3804.4). Total num frames: 3518464. Throughput: 0: 939.9. Samples: 879202. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:09:26,955][00556] Avg episode reward: [(0, '20.252')] [2024-09-05 09:09:27,019][02572] Updated weights for policy 0, policy_version 860 (0.0020) [2024-09-05 09:09:31,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3804.4). Total num frames: 3543040. Throughput: 0: 994.0. Samples: 886348. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:09:31,955][00556] Avg episode reward: [(0, '19.490')] [2024-09-05 09:09:36,953][00556] Fps is (10 sec: 4095.5, 60 sec: 3891.1, 300 sec: 3804.4). Total num frames: 3559424. Throughput: 0: 994.2. Samples: 889024. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:09:36,956][00556] Avg episode reward: [(0, '18.917')] [2024-09-05 09:09:37,635][02572] Updated weights for policy 0, policy_version 870 (0.0038) [2024-09-05 09:09:41,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3804.4). Total num frames: 3575808. Throughput: 0: 945.1. Samples: 893640. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:09:41,954][00556] Avg episode reward: [(0, '19.306')] [2024-09-05 09:09:46,952][00556] Fps is (10 sec: 4096.6, 60 sec: 3959.5, 300 sec: 3804.4). Total num frames: 3600384. Throughput: 0: 977.9. Samples: 900686. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:09:46,954][00556] Avg episode reward: [(0, '19.264')] [2024-09-05 09:09:47,171][02572] Updated weights for policy 0, policy_version 880 (0.0038) [2024-09-05 09:09:51,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3804.5). Total num frames: 3620864. Throughput: 0: 1010.9. Samples: 904234. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:09:51,954][00556] Avg episode reward: [(0, '19.217')] [2024-09-05 09:09:56,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3790.6). Total num frames: 3633152. Throughput: 0: 965.9. Samples: 908652. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:09:56,955][00556] Avg episode reward: [(0, '19.173')] [2024-09-05 09:09:58,834][02572] Updated weights for policy 0, policy_version 890 (0.0041) [2024-09-05 09:10:01,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 3657728. Throughput: 0: 958.0. Samples: 914864. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:10:01,955][00556] Avg episode reward: [(0, '19.644')] [2024-09-05 09:10:06,952][00556] Fps is (10 sec: 4915.1, 60 sec: 4028.0, 300 sec: 3818.3). Total num frames: 3682304. Throughput: 0: 987.8. Samples: 918294. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:10:06,954][00556] Avg episode reward: [(0, '19.645')] [2024-09-05 09:10:08,126][02572] Updated weights for policy 0, policy_version 900 (0.0031) [2024-09-05 09:10:11,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3790.5). Total num frames: 3694592. Throughput: 0: 990.4. Samples: 923768. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:10:11,954][00556] Avg episode reward: [(0, '19.657')] [2024-09-05 09:10:16,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3822.9, 300 sec: 3804.4). Total num frames: 3715072. Throughput: 0: 949.2. Samples: 929062. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:10:16,954][00556] Avg episode reward: [(0, '20.141')] [2024-09-05 09:10:19,372][02572] Updated weights for policy 0, policy_version 910 (0.0025) [2024-09-05 09:10:21,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3959.5, 300 sec: 3804.4). Total num frames: 3735552. Throughput: 0: 966.2. Samples: 932502. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:10:21,955][00556] Avg episode reward: [(0, '21.187')] [2024-09-05 09:10:26,952][00556] Fps is (10 sec: 4096.0, 60 sec: 3959.5, 300 sec: 3804.4). Total num frames: 3756032. Throughput: 0: 1003.2. Samples: 938784. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:10:26,956][00556] Avg episode reward: [(0, '21.733')] [2024-09-05 09:10:30,726][02572] Updated weights for policy 0, policy_version 920 (0.0035) [2024-09-05 09:10:31,952][00556] Fps is (10 sec: 3686.3, 60 sec: 3822.9, 300 sec: 3818.3). Total num frames: 3772416. Throughput: 0: 943.1. Samples: 943126. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:10:31,958][00556] Avg episode reward: [(0, '21.713')] [2024-09-05 09:10:36,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.3, 300 sec: 3804.4). Total num frames: 3792896. Throughput: 0: 942.7. Samples: 946656. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:10:36,959][00556] Avg episode reward: [(0, '21.005')] [2024-09-05 09:10:39,900][02572] Updated weights for policy 0, policy_version 930 (0.0026) [2024-09-05 09:10:41,952][00556] Fps is (10 sec: 4505.8, 60 sec: 4027.7, 300 sec: 3818.3). Total num frames: 3817472. Throughput: 0: 1000.1. Samples: 953658. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:10:41,958][00556] Avg episode reward: [(0, '21.026')] [2024-09-05 09:10:46,952][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3804.4). Total num frames: 3829760. Throughput: 0: 965.3. Samples: 958304. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:10:46,957][00556] Avg episode reward: [(0, '21.165')] [2024-09-05 09:10:51,499][02572] Updated weights for policy 0, policy_version 940 (0.0032) [2024-09-05 09:10:51,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3822.9, 300 sec: 3818.3). Total num frames: 3850240. Throughput: 0: 946.9. Samples: 960906. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:10:51,954][00556] Avg episode reward: [(0, '21.622')] [2024-09-05 09:10:56,952][00556] Fps is (10 sec: 4505.5, 60 sec: 4027.7, 300 sec: 3818.3). Total num frames: 3874816. Throughput: 0: 976.9. Samples: 967728. Policy #0 lag: (min: 0.0, avg: 0.8, max: 2.0) [2024-09-05 09:10:56,959][00556] Avg episode reward: [(0, '22.062')] [2024-09-05 09:11:01,373][02572] Updated weights for policy 0, policy_version 950 (0.0028) [2024-09-05 09:11:01,954][00556] Fps is (10 sec: 4095.2, 60 sec: 3891.1, 300 sec: 3818.4). Total num frames: 3891200. Throughput: 0: 983.8. Samples: 973334. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:11:01,956][00556] Avg episode reward: [(0, '22.559')] [2024-09-05 09:11:01,974][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000950_3891200.pth... [2024-09-05 09:11:02,141][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000726_2973696.pth [2024-09-05 09:11:06,952][00556] Fps is (10 sec: 3276.9, 60 sec: 3754.7, 300 sec: 3818.3). Total num frames: 3907584. Throughput: 0: 953.9. Samples: 975428. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:11:06,960][00556] Avg episode reward: [(0, '22.057')] [2024-09-05 09:11:11,872][02572] Updated weights for policy 0, policy_version 960 (0.0030) [2024-09-05 09:11:11,952][00556] Fps is (10 sec: 4096.8, 60 sec: 3959.5, 300 sec: 3832.2). Total num frames: 3932160. Throughput: 0: 960.3. Samples: 981996. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:11:11,958][00556] Avg episode reward: [(0, '22.809')] [2024-09-05 09:11:16,952][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3818.3). Total num frames: 3952640. Throughput: 0: 1010.2. Samples: 988584. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:11:16,964][00556] Avg episode reward: [(0, '22.974')] [2024-09-05 09:11:21,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3822.9, 300 sec: 3818.4). Total num frames: 3964928. Throughput: 0: 979.0. Samples: 990712. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:11:21,959][00556] Avg episode reward: [(0, '23.266')] [2024-09-05 09:11:23,649][02572] Updated weights for policy 0, policy_version 970 (0.0022) [2024-09-05 09:11:26,952][00556] Fps is (10 sec: 3276.8, 60 sec: 3822.9, 300 sec: 3818.3). Total num frames: 3985408. Throughput: 0: 940.7. Samples: 995988. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:11:26,954][00556] Avg episode reward: [(0, '23.837')] [2024-09-05 09:11:26,962][02559] Saving new best policy, reward=23.837! [2024-09-05 09:11:30,899][02559] Stopping Batcher_0... [2024-09-05 09:11:30,899][02559] Loop batcher_evt_loop terminating... [2024-09-05 09:11:30,899][00556] Component Batcher_0 stopped! [2024-09-05 09:11:30,901][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2024-09-05 09:11:30,964][02572] Weights refcount: 2 0 [2024-09-05 09:11:30,970][02572] Stopping InferenceWorker_p0-w0... [2024-09-05 09:11:30,970][00556] Component InferenceWorker_p0-w0 stopped! [2024-09-05 09:11:30,971][02572] Loop inference_proc0-0_evt_loop terminating... [2024-09-05 09:11:31,090][02559] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000836_3424256.pth [2024-09-05 09:11:31,110][02559] Saving new best policy, reward=24.234! [2024-09-05 09:11:31,274][02559] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2024-09-05 09:11:31,289][00556] Component RolloutWorker_w6 stopped! [2024-09-05 09:11:31,296][02578] Stopping RolloutWorker_w6... [2024-09-05 09:11:31,297][02578] Loop rollout_proc6_evt_loop terminating... [2024-09-05 09:11:31,307][00556] Component RolloutWorker_w4 stopped! [2024-09-05 09:11:31,313][02577] Stopping RolloutWorker_w4... [2024-09-05 09:11:31,315][02577] Loop rollout_proc4_evt_loop terminating... [2024-09-05 09:11:31,321][00556] Component RolloutWorker_w0 stopped! [2024-09-05 09:11:31,327][02573] Stopping RolloutWorker_w0... [2024-09-05 09:11:31,329][02573] Loop rollout_proc0_evt_loop terminating... [2024-09-05 09:11:31,336][00556] Component RolloutWorker_w2 stopped! [2024-09-05 09:11:31,342][02575] Stopping RolloutWorker_w2... [2024-09-05 09:11:31,344][02575] Loop rollout_proc2_evt_loop terminating... [2024-09-05 09:11:31,407][02580] Stopping RolloutWorker_w7... [2024-09-05 09:11:31,407][02580] Loop rollout_proc7_evt_loop terminating... [2024-09-05 09:11:31,411][00556] Component RolloutWorker_w7 stopped! [2024-09-05 09:11:31,444][02574] Stopping RolloutWorker_w1... [2024-09-05 09:11:31,447][00556] Component RolloutWorker_w1 stopped! [2024-09-05 09:11:31,445][02574] Loop rollout_proc1_evt_loop terminating... [2024-09-05 09:11:31,470][02559] Stopping LearnerWorker_p0... [2024-09-05 09:11:31,471][02559] Loop learner_proc0_evt_loop terminating... [2024-09-05 09:11:31,473][00556] Component LearnerWorker_p0 stopped! [2024-09-05 09:11:31,502][02579] Stopping RolloutWorker_w5... [2024-09-05 09:11:31,502][00556] Component RolloutWorker_w5 stopped! [2024-09-05 09:11:31,503][02579] Loop rollout_proc5_evt_loop terminating... [2024-09-05 09:11:31,574][02576] Stopping RolloutWorker_w3... [2024-09-05 09:11:31,574][00556] Component RolloutWorker_w3 stopped! [2024-09-05 09:11:31,578][00556] Waiting for process learner_proc0 to stop... [2024-09-05 09:11:31,579][02576] Loop rollout_proc3_evt_loop terminating... [2024-09-05 09:11:33,034][00556] Waiting for process inference_proc0-0 to join... [2024-09-05 09:11:33,039][00556] Waiting for process rollout_proc0 to join... [2024-09-05 09:11:35,627][00556] Waiting for process rollout_proc1 to join... [2024-09-05 09:11:35,871][00556] Waiting for process rollout_proc2 to join... [2024-09-05 09:11:35,874][00556] Waiting for process rollout_proc3 to join... [2024-09-05 09:11:35,879][00556] Waiting for process rollout_proc4 to join... [2024-09-05 09:11:35,881][00556] Waiting for process rollout_proc5 to join... [2024-09-05 09:11:35,882][00556] Waiting for process rollout_proc6 to join... [2024-09-05 09:11:35,883][00556] Waiting for process rollout_proc7 to join... [2024-09-05 09:11:35,886][00556] Batcher 0 profile tree view: batching: 28.7009, releasing_batches: 0.0292 [2024-09-05 09:11:35,889][00556] InferenceWorker_p0-w0 profile tree view: wait_policy: 0.0001 wait_policy_total: 386.3923 update_model: 9.3406 weight_update: 0.0025 one_step: 0.0123 handle_policy_step: 634.4478 deserialize: 15.2177, stack: 3.3282, obs_to_device_normalize: 129.3269, forward: 336.0333, send_messages: 30.4733 prepare_outputs: 88.4075 to_cpu: 51.2483 [2024-09-05 09:11:35,890][00556] Learner 0 profile tree view: misc: 0.0075, prepare_batch: 14.5544 train: 75.3765 epoch_init: 0.0110, minibatch_init: 0.0067, losses_postprocess: 0.6628, kl_divergence: 0.7560, after_optimizer: 33.5400 calculate_losses: 27.0137 losses_init: 0.0072, forward_head: 1.3314, bptt_initial: 18.0916, tail: 1.1112, advantages_returns: 0.2629, losses: 3.7827 bptt: 2.1088 bptt_forward_core: 2.0185 update: 12.7220 clip: 0.9137 [2024-09-05 09:11:35,892][00556] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.3611, enqueue_policy_requests: 93.7444, env_step: 835.0048, overhead: 13.9005, complete_rollouts: 7.6595 save_policy_outputs: 21.8211 split_output_tensors: 8.8036 [2024-09-05 09:11:35,893][00556] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.3465, enqueue_policy_requests: 96.0715, env_step: 834.0358, overhead: 13.6210, complete_rollouts: 6.8607 save_policy_outputs: 20.8596 split_output_tensors: 8.4140 [2024-09-05 09:11:35,896][00556] Loop Runner_EvtLoop terminating... [2024-09-05 09:11:35,898][00556] Runner profile tree view: main_loop: 1104.8237 [2024-09-05 09:11:35,899][00556] Collected {0: 4005888}, FPS: 3625.8 [2024-09-05 09:11:36,254][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:11:36,255][00556] Overriding arg 'num_workers' with value 1 passed from command line [2024-09-05 09:11:36,259][00556] Adding new argument 'no_render'=True that is not in the saved config file! [2024-09-05 09:11:36,261][00556] Adding new argument 'save_video'=True that is not in the saved config file! [2024-09-05 09:11:36,263][00556] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:11:36,265][00556] Adding new argument 'video_name'=None that is not in the saved config file! [2024-09-05 09:11:36,267][00556] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:11:36,269][00556] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2024-09-05 09:11:36,270][00556] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2024-09-05 09:11:36,271][00556] Adding new argument 'hf_repository'=None that is not in the saved config file! [2024-09-05 09:11:36,273][00556] Adding new argument 'policy_index'=0 that is not in the saved config file! [2024-09-05 09:11:36,274][00556] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2024-09-05 09:11:36,275][00556] Adding new argument 'train_script'=None that is not in the saved config file! [2024-09-05 09:11:36,277][00556] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2024-09-05 09:11:36,278][00556] Using frameskip 1 and render_action_repeat=4 for evaluation [2024-09-05 09:11:36,318][00556] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:11:36,322][00556] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:11:36,325][00556] RunningMeanStd input shape: (1,) [2024-09-05 09:11:36,341][00556] ConvEncoder: input_channels=3 [2024-09-05 09:11:36,456][00556] Conv encoder output size: 512 [2024-09-05 09:11:36,459][00556] Policy head output size: 512 [2024-09-05 09:11:36,639][00556] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2024-09-05 09:11:37,498][00556] Num frames 100... [2024-09-05 09:11:37,626][00556] Num frames 200... [2024-09-05 09:11:37,751][00556] Num frames 300... [2024-09-05 09:11:37,873][00556] Num frames 400... [2024-09-05 09:11:37,996][00556] Num frames 500... [2024-09-05 09:11:38,117][00556] Num frames 600... [2024-09-05 09:11:38,242][00556] Num frames 700... [2024-09-05 09:11:38,367][00556] Num frames 800... [2024-09-05 09:11:38,501][00556] Num frames 900... [2024-09-05 09:11:38,625][00556] Num frames 1000... [2024-09-05 09:11:38,713][00556] Avg episode rewards: #0: 24.240, true rewards: #0: 10.240 [2024-09-05 09:11:38,715][00556] Avg episode reward: 24.240, avg true_objective: 10.240 [2024-09-05 09:11:38,806][00556] Num frames 1100... [2024-09-05 09:11:38,928][00556] Num frames 1200... [2024-09-05 09:11:39,054][00556] Num frames 1300... [2024-09-05 09:11:39,175][00556] Num frames 1400... [2024-09-05 09:11:39,293][00556] Num frames 1500... [2024-09-05 09:11:39,414][00556] Num frames 1600... [2024-09-05 09:11:39,508][00556] Avg episode rewards: #0: 17.660, true rewards: #0: 8.160 [2024-09-05 09:11:39,509][00556] Avg episode reward: 17.660, avg true_objective: 8.160 [2024-09-05 09:11:39,600][00556] Num frames 1700... [2024-09-05 09:11:39,731][00556] Num frames 1800... [2024-09-05 09:11:39,852][00556] Num frames 1900... [2024-09-05 09:11:39,976][00556] Num frames 2000... [2024-09-05 09:11:40,103][00556] Num frames 2100... [2024-09-05 09:11:40,226][00556] Num frames 2200... [2024-09-05 09:11:40,352][00556] Num frames 2300... [2024-09-05 09:11:40,475][00556] Num frames 2400... [2024-09-05 09:11:40,609][00556] Num frames 2500... [2024-09-05 09:11:40,739][00556] Num frames 2600... [2024-09-05 09:11:40,861][00556] Num frames 2700... [2024-09-05 09:11:40,984][00556] Num frames 2800... [2024-09-05 09:11:41,105][00556] Num frames 2900... [2024-09-05 09:11:41,227][00556] Num frames 3000... [2024-09-05 09:11:41,348][00556] Num frames 3100... [2024-09-05 09:11:41,471][00556] Num frames 3200... [2024-09-05 09:11:41,610][00556] Avg episode rewards: #0: 24.547, true rewards: #0: 10.880 [2024-09-05 09:11:41,612][00556] Avg episode reward: 24.547, avg true_objective: 10.880 [2024-09-05 09:11:41,663][00556] Num frames 3300... [2024-09-05 09:11:41,792][00556] Num frames 3400... [2024-09-05 09:11:41,913][00556] Num frames 3500... [2024-09-05 09:11:42,037][00556] Num frames 3600... [2024-09-05 09:11:42,171][00556] Num frames 3700... [2024-09-05 09:11:42,294][00556] Num frames 3800... [2024-09-05 09:11:42,419][00556] Num frames 3900... [2024-09-05 09:11:42,541][00556] Num frames 4000... [2024-09-05 09:11:42,678][00556] Num frames 4100... [2024-09-05 09:11:42,800][00556] Num frames 4200... [2024-09-05 09:11:42,921][00556] Num frames 4300... [2024-09-05 09:11:43,046][00556] Num frames 4400... [2024-09-05 09:11:43,171][00556] Num frames 4500... [2024-09-05 09:11:43,294][00556] Num frames 4600... [2024-09-05 09:11:43,419][00556] Num frames 4700... [2024-09-05 09:11:43,547][00556] Num frames 4800... [2024-09-05 09:11:43,686][00556] Num frames 4900... [2024-09-05 09:11:43,808][00556] Num frames 5000... [2024-09-05 09:11:43,931][00556] Num frames 5100... [2024-09-05 09:11:44,053][00556] Num frames 5200... [2024-09-05 09:11:44,128][00556] Avg episode rewards: #0: 32.040, true rewards: #0: 13.040 [2024-09-05 09:11:44,130][00556] Avg episode reward: 32.040, avg true_objective: 13.040 [2024-09-05 09:11:44,232][00556] Num frames 5300... [2024-09-05 09:11:44,357][00556] Num frames 5400... [2024-09-05 09:11:44,483][00556] Num frames 5500... [2024-09-05 09:11:44,610][00556] Num frames 5600... [2024-09-05 09:11:44,749][00556] Num frames 5700... [2024-09-05 09:11:44,886][00556] Avg episode rewards: #0: 28.336, true rewards: #0: 11.536 [2024-09-05 09:11:44,887][00556] Avg episode reward: 28.336, avg true_objective: 11.536 [2024-09-05 09:11:44,929][00556] Num frames 5800... [2024-09-05 09:11:45,050][00556] Num frames 5900... [2024-09-05 09:11:45,175][00556] Num frames 6000... [2024-09-05 09:11:45,296][00556] Num frames 6100... [2024-09-05 09:11:45,417][00556] Num frames 6200... [2024-09-05 09:11:45,538][00556] Num frames 6300... [2024-09-05 09:11:45,674][00556] Num frames 6400... [2024-09-05 09:11:45,826][00556] Avg episode rewards: #0: 25.953, true rewards: #0: 10.787 [2024-09-05 09:11:45,827][00556] Avg episode reward: 25.953, avg true_objective: 10.787 [2024-09-05 09:11:45,864][00556] Num frames 6500... [2024-09-05 09:11:45,987][00556] Num frames 6600... [2024-09-05 09:11:46,107][00556] Num frames 6700... [2024-09-05 09:11:46,254][00556] Num frames 6800... [2024-09-05 09:11:46,434][00556] Num frames 6900... [2024-09-05 09:11:46,613][00556] Num frames 7000... [2024-09-05 09:11:46,791][00556] Num frames 7100... [2024-09-05 09:11:46,966][00556] Num frames 7200... [2024-09-05 09:11:47,137][00556] Num frames 7300... [2024-09-05 09:11:47,303][00556] Num frames 7400... [2024-09-05 09:11:47,517][00556] Avg episode rewards: #0: 25.566, true rewards: #0: 10.709 [2024-09-05 09:11:47,518][00556] Avg episode reward: 25.566, avg true_objective: 10.709 [2024-09-05 09:11:47,528][00556] Num frames 7500... [2024-09-05 09:11:47,704][00556] Num frames 7600... [2024-09-05 09:11:47,884][00556] Num frames 7700... [2024-09-05 09:11:48,061][00556] Num frames 7800... [2024-09-05 09:11:48,239][00556] Num frames 7900... [2024-09-05 09:11:48,417][00556] Num frames 8000... [2024-09-05 09:11:48,602][00556] Num frames 8100... [2024-09-05 09:11:48,732][00556] Num frames 8200... [2024-09-05 09:11:48,866][00556] Num frames 8300... [2024-09-05 09:11:48,993][00556] Num frames 8400... [2024-09-05 09:11:49,117][00556] Num frames 8500... [2024-09-05 09:11:49,242][00556] Num frames 8600... [2024-09-05 09:11:49,366][00556] Num frames 8700... [2024-09-05 09:11:49,493][00556] Num frames 8800... [2024-09-05 09:11:49,621][00556] Num frames 8900... [2024-09-05 09:11:49,752][00556] Num frames 9000... [2024-09-05 09:11:49,884][00556] Num frames 9100... [2024-09-05 09:11:50,011][00556] Num frames 9200... [2024-09-05 09:11:50,133][00556] Num frames 9300... [2024-09-05 09:11:50,256][00556] Num frames 9400... [2024-09-05 09:11:50,413][00556] Avg episode rewards: #0: 28.600, true rewards: #0: 11.850 [2024-09-05 09:11:50,414][00556] Avg episode reward: 28.600, avg true_objective: 11.850 [2024-09-05 09:11:50,442][00556] Num frames 9500... [2024-09-05 09:11:50,568][00556] Num frames 9600... [2024-09-05 09:11:50,699][00556] Num frames 9700... [2024-09-05 09:11:50,822][00556] Num frames 9800... [2024-09-05 09:11:50,953][00556] Num frames 9900... [2024-09-05 09:11:51,079][00556] Num frames 10000... [2024-09-05 09:11:51,202][00556] Num frames 10100... [2024-09-05 09:11:51,324][00556] Num frames 10200... [2024-09-05 09:11:51,448][00556] Num frames 10300... [2024-09-05 09:11:51,572][00556] Num frames 10400... [2024-09-05 09:11:51,710][00556] Num frames 10500... [2024-09-05 09:11:51,834][00556] Num frames 10600... [2024-09-05 09:11:51,969][00556] Num frames 10700... [2024-09-05 09:11:52,099][00556] Num frames 10800... [2024-09-05 09:11:52,228][00556] Num frames 10900... [2024-09-05 09:11:52,357][00556] Num frames 11000... [2024-09-05 09:11:52,482][00556] Num frames 11100... [2024-09-05 09:11:52,607][00556] Num frames 11200... [2024-09-05 09:11:52,740][00556] Num frames 11300... [2024-09-05 09:11:52,889][00556] Num frames 11400... [2024-09-05 09:11:53,043][00556] Num frames 11500... [2024-09-05 09:11:53,197][00556] Avg episode rewards: #0: 31.755, true rewards: #0: 12.867 [2024-09-05 09:11:53,198][00556] Avg episode reward: 31.755, avg true_objective: 12.867 [2024-09-05 09:11:53,226][00556] Num frames 11600... [2024-09-05 09:11:53,348][00556] Num frames 11700... [2024-09-05 09:11:53,474][00556] Num frames 11800... [2024-09-05 09:11:53,599][00556] Num frames 11900... [2024-09-05 09:11:53,737][00556] Num frames 12000... [2024-09-05 09:11:53,858][00556] Num frames 12100... [2024-09-05 09:11:54,029][00556] Avg episode rewards: #0: 29.488, true rewards: #0: 12.188 [2024-09-05 09:11:54,030][00556] Avg episode reward: 29.488, avg true_objective: 12.188 [2024-09-05 09:13:13,015][00556] Replay video saved to /content/train_dir/default_experiment/replay.mp4! [2024-09-05 09:18:45,792][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:18:45,794][00556] Overriding arg 'num_workers' with value 1 passed from command line [2024-09-05 09:18:45,796][00556] Adding new argument 'no_render'=True that is not in the saved config file! [2024-09-05 09:18:45,798][00556] Adding new argument 'save_video'=True that is not in the saved config file! [2024-09-05 09:18:45,800][00556] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:18:45,802][00556] Adding new argument 'video_name'=None that is not in the saved config file! [2024-09-05 09:18:45,803][00556] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2024-09-05 09:18:45,805][00556] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2024-09-05 09:18:45,806][00556] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2024-09-05 09:18:45,807][00556] Adding new argument 'hf_repository'='neeldevenshah/rl_course_vizdoom_health_gathering_supreme' that is not in the saved config file! [2024-09-05 09:18:45,808][00556] Adding new argument 'policy_index'=0 that is not in the saved config file! [2024-09-05 09:18:45,809][00556] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2024-09-05 09:18:45,810][00556] Adding new argument 'train_script'=None that is not in the saved config file! [2024-09-05 09:18:45,811][00556] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2024-09-05 09:18:45,812][00556] Using frameskip 1 and render_action_repeat=4 for evaluation [2024-09-05 09:18:45,842][00556] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:18:45,843][00556] RunningMeanStd input shape: (1,) [2024-09-05 09:18:45,856][00556] ConvEncoder: input_channels=3 [2024-09-05 09:18:45,894][00556] Conv encoder output size: 512 [2024-09-05 09:18:45,895][00556] Policy head output size: 512 [2024-09-05 09:18:45,914][00556] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2024-09-05 09:18:46,357][00556] Num frames 100... [2024-09-05 09:18:46,482][00556] Num frames 200... [2024-09-05 09:18:46,603][00556] Num frames 300... [2024-09-05 09:18:46,726][00556] Avg episode rewards: #0: 4.520, true rewards: #0: 3.520 [2024-09-05 09:18:46,728][00556] Avg episode reward: 4.520, avg true_objective: 3.520 [2024-09-05 09:18:46,799][00556] Num frames 400... [2024-09-05 09:18:46,926][00556] Num frames 500... [2024-09-05 09:18:47,066][00556] Num frames 600... [2024-09-05 09:18:47,188][00556] Num frames 700... [2024-09-05 09:18:47,311][00556] Num frames 800... [2024-09-05 09:18:47,440][00556] Num frames 900... [2024-09-05 09:18:47,568][00556] Num frames 1000... [2024-09-05 09:18:47,696][00556] Num frames 1100... [2024-09-05 09:18:47,818][00556] Num frames 1200... [2024-09-05 09:18:47,941][00556] Num frames 1300... [2024-09-05 09:18:48,081][00556] Num frames 1400... [2024-09-05 09:18:48,203][00556] Num frames 1500... [2024-09-05 09:18:48,326][00556] Num frames 1600... [2024-09-05 09:18:48,459][00556] Num frames 1700... [2024-09-05 09:18:48,585][00556] Num frames 1800... [2024-09-05 09:18:48,718][00556] Num frames 1900... [2024-09-05 09:18:48,798][00556] Avg episode rewards: #0: 24.600, true rewards: #0: 9.600 [2024-09-05 09:18:48,799][00556] Avg episode reward: 24.600, avg true_objective: 9.600 [2024-09-05 09:18:48,903][00556] Num frames 2000... [2024-09-05 09:18:49,027][00556] Num frames 2100... [2024-09-05 09:18:49,154][00556] Num frames 2200... [2024-09-05 09:18:49,286][00556] Num frames 2300... [2024-09-05 09:18:49,412][00556] Num frames 2400... [2024-09-05 09:18:49,538][00556] Num frames 2500... [2024-09-05 09:18:49,662][00556] Num frames 2600... [2024-09-05 09:18:49,824][00556] Num frames 2700... [2024-09-05 09:18:50,004][00556] Num frames 2800... [2024-09-05 09:18:50,186][00556] Num frames 2900... [2024-09-05 09:18:50,360][00556] Num frames 3000... [2024-09-05 09:18:50,536][00556] Num frames 3100... [2024-09-05 09:18:50,706][00556] Num frames 3200... [2024-09-05 09:18:50,876][00556] Num frames 3300... [2024-09-05 09:18:51,053][00556] Num frames 3400... [2024-09-05 09:18:51,237][00556] Num frames 3500... [2024-09-05 09:18:51,410][00556] Num frames 3600... [2024-09-05 09:18:51,593][00556] Num frames 3700... [2024-09-05 09:18:51,783][00556] Num frames 3800... [2024-09-05 09:18:51,964][00556] Num frames 3900... [2024-09-05 09:18:52,044][00556] Avg episode rewards: #0: 33.373, true rewards: #0: 13.040 [2024-09-05 09:18:52,046][00556] Avg episode reward: 33.373, avg true_objective: 13.040 [2024-09-05 09:18:52,231][00556] Num frames 4000... [2024-09-05 09:18:52,385][00556] Num frames 4100... [2024-09-05 09:18:52,509][00556] Num frames 4200... [2024-09-05 09:18:52,663][00556] Num frames 4300... [2024-09-05 09:18:52,802][00556] Num frames 4400... [2024-09-05 09:18:52,925][00556] Num frames 4500... [2024-09-05 09:18:53,052][00556] Num frames 4600... [2024-09-05 09:18:53,211][00556] Avg episode rewards: #0: 28.700, true rewards: #0: 11.700 [2024-09-05 09:18:53,212][00556] Avg episode reward: 28.700, avg true_objective: 11.700 [2024-09-05 09:18:53,243][00556] Num frames 4700... [2024-09-05 09:18:53,395][00556] Num frames 4800... [2024-09-05 09:18:53,519][00556] Num frames 4900... [2024-09-05 09:18:53,647][00556] Num frames 5000... [2024-09-05 09:18:53,784][00556] Num frames 5100... [2024-09-05 09:18:53,905][00556] Num frames 5200... [2024-09-05 09:18:54,033][00556] Num frames 5300... [2024-09-05 09:18:54,189][00556] Avg episode rewards: #0: 25.168, true rewards: #0: 10.768 [2024-09-05 09:18:54,190][00556] Avg episode reward: 25.168, avg true_objective: 10.768 [2024-09-05 09:18:54,214][00556] Num frames 5400... [2024-09-05 09:18:54,347][00556] Num frames 5500... [2024-09-05 09:18:54,473][00556] Num frames 5600... [2024-09-05 09:18:54,599][00556] Num frames 5700... [2024-09-05 09:18:54,728][00556] Num frames 5800... [2024-09-05 09:18:54,851][00556] Num frames 5900... [2024-09-05 09:18:54,975][00556] Num frames 6000... [2024-09-05 09:18:55,102][00556] Avg episode rewards: #0: 23.426, true rewards: #0: 10.093 [2024-09-05 09:18:55,104][00556] Avg episode reward: 23.426, avg true_objective: 10.093 [2024-09-05 09:18:55,161][00556] Num frames 6100... [2024-09-05 09:18:55,289][00556] Num frames 6200... [2024-09-05 09:18:55,432][00556] Num frames 6300... [2024-09-05 09:18:55,559][00556] Num frames 6400... [2024-09-05 09:18:55,665][00556] Avg episode rewards: #0: 21.343, true rewards: #0: 9.200 [2024-09-05 09:18:55,667][00556] Avg episode reward: 21.343, avg true_objective: 9.200 [2024-09-05 09:18:55,745][00556] Num frames 6500... [2024-09-05 09:18:55,868][00556] Num frames 6600... [2024-09-05 09:18:55,996][00556] Num frames 6700... [2024-09-05 09:18:56,127][00556] Num frames 6800... [2024-09-05 09:18:56,287][00556] Num frames 6900... [2024-09-05 09:18:56,423][00556] Num frames 7000... [2024-09-05 09:18:56,551][00556] Num frames 7100... [2024-09-05 09:18:56,690][00556] Num frames 7200... [2024-09-05 09:18:56,816][00556] Num frames 7300... [2024-09-05 09:18:56,914][00556] Avg episode rewards: #0: 20.670, true rewards: #0: 9.170 [2024-09-05 09:18:56,915][00556] Avg episode reward: 20.670, avg true_objective: 9.170 [2024-09-05 09:18:56,999][00556] Num frames 7400... [2024-09-05 09:18:57,126][00556] Num frames 7500... [2024-09-05 09:18:57,251][00556] Num frames 7600... [2024-09-05 09:18:57,385][00556] Num frames 7700... [2024-09-05 09:18:57,513][00556] Num frames 7800... [2024-09-05 09:18:57,642][00556] Num frames 7900... [2024-09-05 09:18:57,783][00556] Num frames 8000... [2024-09-05 09:18:57,903][00556] Num frames 8100... [2024-09-05 09:18:58,027][00556] Num frames 8200... [2024-09-05 09:18:58,150][00556] Num frames 8300... [2024-09-05 09:18:58,277][00556] Num frames 8400... [2024-09-05 09:18:58,363][00556] Avg episode rewards: #0: 20.693, true rewards: #0: 9.360 [2024-09-05 09:18:58,366][00556] Avg episode reward: 20.693, avg true_objective: 9.360 [2024-09-05 09:18:58,477][00556] Num frames 8500... [2024-09-05 09:18:58,604][00556] Num frames 8600... [2024-09-05 09:18:58,734][00556] Num frames 8700... [2024-09-05 09:18:58,859][00556] Num frames 8800... [2024-09-05 09:18:58,982][00556] Num frames 8900... [2024-09-05 09:18:59,106][00556] Num frames 9000... [2024-09-05 09:18:59,172][00556] Avg episode rewards: #0: 19.908, true rewards: #0: 9.008 [2024-09-05 09:18:59,173][00556] Avg episode reward: 19.908, avg true_objective: 9.008 [2024-09-05 09:19:58,814][00556] Replay video saved to /content/train_dir/default_experiment/replay.mp4! [2024-09-05 09:20:05,980][00556] The model has been pushed to https://huggingface.co./neeldevenshah/rl_course_vizdoom_health_gathering_supreme [2024-09-05 09:22:40,327][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:22:40,329][00556] Overriding arg 'train_for_env_steps' with value 2000000 passed from command line [2024-09-05 09:22:40,340][00556] Experiment dir /content/train_dir/default_experiment already exists! [2024-09-05 09:22:40,342][00556] Resuming existing experiment from /content/train_dir/default_experiment... [2024-09-05 09:22:40,346][00556] Weights and Biases integration disabled [2024-09-05 09:22:40,352][00556] Environment var CUDA_VISIBLE_DEVICES is 0 [2024-09-05 09:22:42,659][00556] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/content/train_dir restart_behavior=resume device=gpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=1024 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=2000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=unknown git_repo_name=not a git repository [2024-09-05 09:22:42,661][00556] Saving configuration to /content/train_dir/default_experiment/config.json... [2024-09-05 09:22:42,665][00556] Rollout worker 0 uses device cpu [2024-09-05 09:22:42,668][00556] Rollout worker 1 uses device cpu [2024-09-05 09:22:42,670][00556] Rollout worker 2 uses device cpu [2024-09-05 09:22:42,672][00556] Rollout worker 3 uses device cpu [2024-09-05 09:22:42,673][00556] Rollout worker 4 uses device cpu [2024-09-05 09:22:42,674][00556] Rollout worker 5 uses device cpu [2024-09-05 09:22:42,675][00556] Rollout worker 6 uses device cpu [2024-09-05 09:22:42,676][00556] Rollout worker 7 uses device cpu [2024-09-05 09:22:42,751][00556] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:22:42,752][00556] InferenceWorker_p0-w0: min num requests: 2 [2024-09-05 09:22:42,918][00556] Starting all processes... [2024-09-05 09:22:42,921][00556] Starting process learner_proc0 [2024-09-05 09:22:42,968][00556] Starting all processes... [2024-09-05 09:22:42,973][00556] Starting process inference_proc0-0 [2024-09-05 09:22:42,974][00556] Starting process rollout_proc0 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc1 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc2 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc3 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc4 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc5 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc6 [2024-09-05 09:22:42,976][00556] Starting process rollout_proc7 [2024-09-05 09:22:59,230][13525] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:22:59,231][13525] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 [2024-09-05 09:22:59,317][13525] Num visible devices: 1 [2024-09-05 09:22:59,345][13525] Starting seed is not provided [2024-09-05 09:22:59,346][13525] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:22:59,347][13525] Initializing actor-critic model on device cuda:0 [2024-09-05 09:22:59,348][13525] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:22:59,349][13525] RunningMeanStd input shape: (1,) [2024-09-05 09:22:59,419][13525] ConvEncoder: input_channels=3 [2024-09-05 09:22:59,581][13541] Worker 1 uses CPU cores [1] [2024-09-05 09:22:59,601][13540] Worker 2 uses CPU cores [0] [2024-09-05 09:22:59,607][13545] Worker 5 uses CPU cores [1] [2024-09-05 09:22:59,667][13539] Worker 0 uses CPU cores [0] [2024-09-05 09:22:59,687][13543] Worker 4 uses CPU cores [0] [2024-09-05 09:22:59,699][13542] Worker 3 uses CPU cores [1] [2024-09-05 09:22:59,707][13546] Worker 6 uses CPU cores [0] [2024-09-05 09:22:59,734][13538] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:22:59,735][13538] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 [2024-09-05 09:22:59,778][13538] Num visible devices: 1 [2024-09-05 09:22:59,807][13544] Worker 7 uses CPU cores [1] [2024-09-05 09:22:59,844][13525] Conv encoder output size: 512 [2024-09-05 09:22:59,844][13525] Policy head output size: 512 [2024-09-05 09:22:59,861][13525] Created Actor Critic model with architecture: [2024-09-05 09:22:59,861][13525] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2024-09-05 09:22:59,992][13525] Using optimizer [2024-09-05 09:23:00,592][13525] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... [2024-09-05 09:23:00,628][13525] Loading model from checkpoint [2024-09-05 09:23:00,630][13525] Loaded experiment state at self.train_step=978, self.env_steps=4005888 [2024-09-05 09:23:00,630][13525] Initialized policy 0 weights for model version 978 [2024-09-05 09:23:00,634][13525] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:23:00,643][13525] LearnerWorker_p0 finished initialization! [2024-09-05 09:23:00,772][13538] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:23:00,774][13538] RunningMeanStd input shape: (1,) [2024-09-05 09:23:00,797][13538] ConvEncoder: input_channels=3 [2024-09-05 09:23:00,902][13538] Conv encoder output size: 512 [2024-09-05 09:23:00,903][13538] Policy head output size: 512 [2024-09-05 09:23:00,956][00556] Inference worker 0-0 is ready! [2024-09-05 09:23:00,958][00556] All inference workers are ready! Signal rollout workers to start! [2024-09-05 09:23:01,269][13540] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,280][13544] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,285][13546] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,287][13545] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,287][13539] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,290][13541] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,292][13543] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:01,413][13542] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:23:02,350][13544] Decorrelating experience for 0 frames... [2024-09-05 09:23:02,349][13541] Decorrelating experience for 0 frames... [2024-09-05 09:23:02,742][00556] Heartbeat connected on Batcher_0 [2024-09-05 09:23:02,747][00556] Heartbeat connected on LearnerWorker_p0 [2024-09-05 09:23:02,778][00556] Heartbeat connected on InferenceWorker_p0-w0 [2024-09-05 09:23:03,046][13540] Decorrelating experience for 0 frames... [2024-09-05 09:23:03,058][13546] Decorrelating experience for 0 frames... [2024-09-05 09:23:03,064][13539] Decorrelating experience for 0 frames... [2024-09-05 09:23:03,072][13543] Decorrelating experience for 0 frames... [2024-09-05 09:23:03,115][13541] Decorrelating experience for 32 frames... [2024-09-05 09:23:03,296][13542] Decorrelating experience for 0 frames... [2024-09-05 09:23:04,066][13544] Decorrelating experience for 32 frames... [2024-09-05 09:23:04,144][13545] Decorrelating experience for 0 frames... [2024-09-05 09:23:04,523][13546] Decorrelating experience for 32 frames... [2024-09-05 09:23:04,526][13539] Decorrelating experience for 32 frames... [2024-09-05 09:23:04,634][13540] Decorrelating experience for 32 frames... [2024-09-05 09:23:05,099][13543] Decorrelating experience for 32 frames... [2024-09-05 09:23:05,356][00556] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 4005888. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 09:23:05,493][13542] Decorrelating experience for 32 frames... [2024-09-05 09:23:05,676][13545] Decorrelating experience for 32 frames... [2024-09-05 09:23:06,764][13541] Decorrelating experience for 64 frames... [2024-09-05 09:23:07,325][13546] Decorrelating experience for 64 frames... [2024-09-05 09:23:07,343][13544] Decorrelating experience for 64 frames... [2024-09-05 09:23:07,467][13540] Decorrelating experience for 64 frames... [2024-09-05 09:23:07,888][13543] Decorrelating experience for 64 frames... [2024-09-05 09:23:08,078][13539] Decorrelating experience for 64 frames... [2024-09-05 09:23:08,111][13542] Decorrelating experience for 64 frames... [2024-09-05 09:23:08,779][13545] Decorrelating experience for 64 frames... [2024-09-05 09:23:09,261][13546] Decorrelating experience for 96 frames... [2024-09-05 09:23:09,361][13540] Decorrelating experience for 96 frames... [2024-09-05 09:23:09,562][00556] Heartbeat connected on RolloutWorker_w6 [2024-09-05 09:23:09,818][00556] Heartbeat connected on RolloutWorker_w2 [2024-09-05 09:23:10,166][13539] Decorrelating experience for 96 frames... [2024-09-05 09:23:10,355][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4005888. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 09:23:10,607][00556] Heartbeat connected on RolloutWorker_w0 [2024-09-05 09:23:10,671][13541] Decorrelating experience for 96 frames... [2024-09-05 09:23:11,046][13542] Decorrelating experience for 96 frames... [2024-09-05 09:23:11,122][00556] Heartbeat connected on RolloutWorker_w1 [2024-09-05 09:23:11,396][00556] Heartbeat connected on RolloutWorker_w3 [2024-09-05 09:23:11,696][13544] Decorrelating experience for 96 frames... [2024-09-05 09:23:11,833][13543] Decorrelating experience for 96 frames... [2024-09-05 09:23:12,278][00556] Heartbeat connected on RolloutWorker_w7 [2024-09-05 09:23:12,492][00556] Heartbeat connected on RolloutWorker_w4 [2024-09-05 09:23:13,407][13545] Decorrelating experience for 96 frames... [2024-09-05 09:23:14,057][00556] Heartbeat connected on RolloutWorker_w5 [2024-09-05 09:23:15,353][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4005888. Throughput: 0: 119.0. Samples: 1190. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 09:23:15,355][00556] Avg episode reward: [(0, '4.190')] [2024-09-05 09:23:16,207][13525] Signal inference workers to stop experience collection... [2024-09-05 09:23:16,227][13538] InferenceWorker_p0-w0: stopping experience collection [2024-09-05 09:23:17,702][13525] Signal inference workers to resume experience collection... [2024-09-05 09:23:17,726][00556] Component Batcher_0 stopped! [2024-09-05 09:23:17,733][13525] Stopping Batcher_0... [2024-09-05 09:23:17,734][13525] Loop batcher_evt_loop terminating... [2024-09-05 09:23:17,771][13538] Weights refcount: 2 0 [2024-09-05 09:23:17,775][00556] Component InferenceWorker_p0-w0 stopped! [2024-09-05 09:23:17,780][13538] Stopping InferenceWorker_p0-w0... [2024-09-05 09:23:17,781][13538] Loop inference_proc0-0_evt_loop terminating... [2024-09-05 09:23:18,026][00556] Component RolloutWorker_w3 stopped! [2024-09-05 09:23:18,032][13542] Stopping RolloutWorker_w3... [2024-09-05 09:23:18,034][13542] Loop rollout_proc3_evt_loop terminating... [2024-09-05 09:23:18,051][00556] Component RolloutWorker_w1 stopped! [2024-09-05 09:23:18,055][13541] Stopping RolloutWorker_w1... [2024-09-05 09:23:18,057][13541] Loop rollout_proc1_evt_loop terminating... [2024-09-05 09:23:18,066][00556] Component RolloutWorker_w7 stopped! [2024-09-05 09:23:18,070][13544] Stopping RolloutWorker_w7... [2024-09-05 09:23:18,075][13544] Loop rollout_proc7_evt_loop terminating... [2024-09-05 09:23:18,090][00556] Component RolloutWorker_w5 stopped! [2024-09-05 09:23:18,095][13545] Stopping RolloutWorker_w5... [2024-09-05 09:23:18,096][13545] Loop rollout_proc5_evt_loop terminating... [2024-09-05 09:23:18,163][13539] Stopping RolloutWorker_w0... [2024-09-05 09:23:18,163][13539] Loop rollout_proc0_evt_loop terminating... [2024-09-05 09:23:18,166][13543] Stopping RolloutWorker_w4... [2024-09-05 09:23:18,170][13543] Loop rollout_proc4_evt_loop terminating... [2024-09-05 09:23:18,163][00556] Component RolloutWorker_w0 stopped! [2024-09-05 09:23:18,171][00556] Component RolloutWorker_w4 stopped! [2024-09-05 09:23:18,205][13546] Stopping RolloutWorker_w6... [2024-09-05 09:23:18,206][13546] Loop rollout_proc6_evt_loop terminating... [2024-09-05 09:23:18,205][00556] Component RolloutWorker_w6 stopped! [2024-09-05 09:23:18,211][13540] Stopping RolloutWorker_w2... [2024-09-05 09:23:18,211][13540] Loop rollout_proc2_evt_loop terminating... [2024-09-05 09:23:18,211][00556] Component RolloutWorker_w2 stopped! [2024-09-05 09:23:18,583][13525] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000980_4014080.pth... [2024-09-05 09:23:18,725][13525] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000950_3891200.pth [2024-09-05 09:23:18,813][13525] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000981_4018176.pth... [2024-09-05 09:23:18,994][13525] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth [2024-09-05 09:23:19,030][13525] Stopping LearnerWorker_p0... [2024-09-05 09:23:19,031][13525] Loop learner_proc0_evt_loop terminating... [2024-09-05 09:23:19,032][00556] Component LearnerWorker_p0 stopped! [2024-09-05 09:23:19,035][00556] Waiting for process learner_proc0 to stop... [2024-09-05 09:23:20,461][00556] Waiting for process inference_proc0-0 to join... [2024-09-05 09:23:20,464][00556] Waiting for process rollout_proc0 to join... [2024-09-05 09:23:22,069][00556] Waiting for process rollout_proc1 to join... [2024-09-05 09:23:22,076][00556] Waiting for process rollout_proc2 to join... [2024-09-05 09:23:22,077][00556] Waiting for process rollout_proc3 to join... [2024-09-05 09:23:22,079][00556] Waiting for process rollout_proc4 to join... [2024-09-05 09:23:22,082][00556] Waiting for process rollout_proc5 to join... [2024-09-05 09:23:22,084][00556] Waiting for process rollout_proc6 to join... [2024-09-05 09:23:22,085][00556] Waiting for process rollout_proc7 to join... [2024-09-05 09:23:22,087][00556] Batcher 0 profile tree view: batching: 0.8961, releasing_batches: 0.0236 [2024-09-05 09:23:22,088][00556] InferenceWorker_p0-w0 profile tree view: update_model: 0.0191 wait_policy: 0.0053 wait_policy_total: 10.7084 one_step: 0.0030 handle_policy_step: 4.1885 deserialize: 0.0672, stack: 0.0200, obs_to_device_normalize: 0.7834, forward: 2.7947, send_messages: 0.1187 prepare_outputs: 0.2949 to_cpu: 0.1654 [2024-09-05 09:23:22,090][00556] Learner 0 profile tree view: misc: 0.0000, prepare_batch: 2.5375 train: 2.8481 epoch_init: 0.0000, minibatch_init: 0.0000, losses_postprocess: 0.0005, kl_divergence: 0.0105, after_optimizer: 0.0580 calculate_losses: 1.4088 losses_init: 0.0000, forward_head: 0.4098, bptt_initial: 0.8584, tail: 0.0752, advantages_returns: 0.0011, losses: 0.0453 bptt: 0.0081 bptt_forward_core: 0.0079 update: 1.3689 clip: 0.0561 [2024-09-05 09:23:22,091][00556] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.0009, enqueue_policy_requests: 0.5479, env_step: 3.3452, overhead: 0.0306, complete_rollouts: 0.0128 save_policy_outputs: 0.0620 split_output_tensors: 0.0194 [2024-09-05 09:23:22,093][00556] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.0011, enqueue_policy_requests: 0.4878, env_step: 2.5696, overhead: 0.0508, complete_rollouts: 0.0444 save_policy_outputs: 0.0733 split_output_tensors: 0.0311 [2024-09-05 09:23:22,095][00556] Loop Runner_EvtLoop terminating... [2024-09-05 09:23:22,097][00556] Runner profile tree view: main_loop: 39.1793 [2024-09-05 09:23:22,099][00556] Collected {0: 4018176}, FPS: 313.6 [2024-09-05 09:23:22,118][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:23:22,129][00556] Overriding arg 'num_workers' with value 1 passed from command line [2024-09-05 09:23:22,131][00556] Adding new argument 'no_render'=True that is not in the saved config file! [2024-09-05 09:23:22,132][00556] Adding new argument 'save_video'=True that is not in the saved config file! [2024-09-05 09:23:22,133][00556] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:23:22,136][00556] Adding new argument 'video_name'=None that is not in the saved config file! [2024-09-05 09:23:22,137][00556] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:23:22,139][00556] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2024-09-05 09:23:22,140][00556] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2024-09-05 09:23:22,141][00556] Adding new argument 'hf_repository'=None that is not in the saved config file! [2024-09-05 09:23:22,143][00556] Adding new argument 'policy_index'=0 that is not in the saved config file! [2024-09-05 09:23:22,144][00556] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2024-09-05 09:23:22,145][00556] Adding new argument 'train_script'=None that is not in the saved config file! [2024-09-05 09:23:22,146][00556] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2024-09-05 09:23:22,147][00556] Using frameskip 1 and render_action_repeat=4 for evaluation [2024-09-05 09:23:22,179][00556] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:23:22,180][00556] RunningMeanStd input shape: (1,) [2024-09-05 09:23:22,193][00556] ConvEncoder: input_channels=3 [2024-09-05 09:23:22,231][00556] Conv encoder output size: 512 [2024-09-05 09:23:22,232][00556] Policy head output size: 512 [2024-09-05 09:23:22,255][00556] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000981_4018176.pth... [2024-09-05 09:23:22,914][00556] Num frames 100... [2024-09-05 09:23:23,082][00556] Num frames 200... [2024-09-05 09:23:23,252][00556] Num frames 300... [2024-09-05 09:23:23,421][00556] Num frames 400... [2024-09-05 09:23:23,589][00556] Num frames 500... [2024-09-05 09:23:23,771][00556] Num frames 600... [2024-09-05 09:23:24,002][00556] Avg episode rewards: #0: 14.880, true rewards: #0: 6.880 [2024-09-05 09:23:24,004][00556] Avg episode reward: 14.880, avg true_objective: 6.880 [2024-09-05 09:23:24,031][00556] Num frames 700... [2024-09-05 09:23:24,204][00556] Num frames 800... [2024-09-05 09:23:24,381][00556] Num frames 900... [2024-09-05 09:23:24,575][00556] Num frames 1000... [2024-09-05 09:23:24,759][00556] Num frames 1100... [2024-09-05 09:23:24,941][00556] Num frames 1200... [2024-09-05 09:23:25,104][00556] Num frames 1300... [2024-09-05 09:23:25,228][00556] Num frames 1400... [2024-09-05 09:23:25,355][00556] Num frames 1500... [2024-09-05 09:23:25,475][00556] Num frames 1600... [2024-09-05 09:23:25,600][00556] Num frames 1700... [2024-09-05 09:23:25,732][00556] Num frames 1800... [2024-09-05 09:23:25,864][00556] Num frames 1900... [2024-09-05 09:23:25,995][00556] Num frames 2000... [2024-09-05 09:23:26,118][00556] Num frames 2100... [2024-09-05 09:23:26,245][00556] Num frames 2200... [2024-09-05 09:23:26,379][00556] Num frames 2300... [2024-09-05 09:23:26,507][00556] Num frames 2400... [2024-09-05 09:23:26,632][00556] Num frames 2500... [2024-09-05 09:23:26,763][00556] Num frames 2600... [2024-09-05 09:23:26,898][00556] Num frames 2700... [2024-09-05 09:23:27,069][00556] Avg episode rewards: #0: 35.440, true rewards: #0: 13.940 [2024-09-05 09:23:27,070][00556] Avg episode reward: 35.440, avg true_objective: 13.940 [2024-09-05 09:23:27,088][00556] Num frames 2800... [2024-09-05 09:23:27,210][00556] Num frames 2900... [2024-09-05 09:23:27,337][00556] Num frames 3000... [2024-09-05 09:23:27,460][00556] Num frames 3100... [2024-09-05 09:23:27,589][00556] Num frames 3200... [2024-09-05 09:23:27,726][00556] Num frames 3300... [2024-09-05 09:23:27,849][00556] Num frames 3400... [2024-09-05 09:23:27,983][00556] Num frames 3500... [2024-09-05 09:23:28,106][00556] Avg episode rewards: #0: 29.853, true rewards: #0: 11.853 [2024-09-05 09:23:28,108][00556] Avg episode reward: 29.853, avg true_objective: 11.853 [2024-09-05 09:23:28,163][00556] Num frames 3600... [2024-09-05 09:23:28,290][00556] Num frames 3700... [2024-09-05 09:23:28,415][00556] Num frames 3800... [2024-09-05 09:23:28,540][00556] Num frames 3900... [2024-09-05 09:23:28,668][00556] Num frames 4000... [2024-09-05 09:23:28,792][00556] Num frames 4100... [2024-09-05 09:23:28,916][00556] Num frames 4200... [2024-09-05 09:23:29,048][00556] Num frames 4300... [2024-09-05 09:23:29,173][00556] Num frames 4400... [2024-09-05 09:23:29,324][00556] Num frames 4500... [2024-09-05 09:23:29,452][00556] Num frames 4600... [2024-09-05 09:23:29,580][00556] Num frames 4700... [2024-09-05 09:23:29,712][00556] Num frames 4800... [2024-09-05 09:23:29,837][00556] Num frames 4900... [2024-09-05 09:23:29,889][00556] Avg episode rewards: #0: 29.750, true rewards: #0: 12.250 [2024-09-05 09:23:29,891][00556] Avg episode reward: 29.750, avg true_objective: 12.250 [2024-09-05 09:23:30,024][00556] Num frames 5000... [2024-09-05 09:23:30,149][00556] Num frames 5100... [2024-09-05 09:23:30,272][00556] Num frames 5200... [2024-09-05 09:23:30,400][00556] Num frames 5300... [2024-09-05 09:23:30,524][00556] Num frames 5400... [2024-09-05 09:23:30,654][00556] Num frames 5500... [2024-09-05 09:23:30,783][00556] Num frames 5600... [2024-09-05 09:23:30,844][00556] Avg episode rewards: #0: 27.008, true rewards: #0: 11.208 [2024-09-05 09:23:30,845][00556] Avg episode reward: 27.008, avg true_objective: 11.208 [2024-09-05 09:23:30,968][00556] Num frames 5700... [2024-09-05 09:23:31,101][00556] Num frames 5800... [2024-09-05 09:23:31,228][00556] Num frames 5900... [2024-09-05 09:23:31,353][00556] Num frames 6000... [2024-09-05 09:23:31,480][00556] Num frames 6100... [2024-09-05 09:23:31,603][00556] Num frames 6200... [2024-09-05 09:23:31,738][00556] Num frames 6300... [2024-09-05 09:23:31,863][00556] Num frames 6400... [2024-09-05 09:23:31,938][00556] Avg episode rewards: #0: 25.858, true rewards: #0: 10.692 [2024-09-05 09:23:31,939][00556] Avg episode reward: 25.858, avg true_objective: 10.692 [2024-09-05 09:23:32,054][00556] Num frames 6500... [2024-09-05 09:23:32,185][00556] Num frames 6600... [2024-09-05 09:23:32,312][00556] Num frames 6700... [2024-09-05 09:23:32,435][00556] Num frames 6800... [2024-09-05 09:23:32,560][00556] Num frames 6900... [2024-09-05 09:23:32,690][00556] Num frames 7000... [2024-09-05 09:23:32,812][00556] Num frames 7100... [2024-09-05 09:23:32,969][00556] Avg episode rewards: #0: 24.261, true rewards: #0: 10.261 [2024-09-05 09:23:32,971][00556] Avg episode reward: 24.261, avg true_objective: 10.261 [2024-09-05 09:23:32,995][00556] Num frames 7200... [2024-09-05 09:23:33,128][00556] Num frames 7300... [2024-09-05 09:23:33,249][00556] Num frames 7400... [2024-09-05 09:23:33,376][00556] Num frames 7500... [2024-09-05 09:23:33,500][00556] Num frames 7600... [2024-09-05 09:23:33,622][00556] Num frames 7700... [2024-09-05 09:23:33,755][00556] Num frames 7800... [2024-09-05 09:23:33,877][00556] Num frames 7900... [2024-09-05 09:23:33,999][00556] Num frames 8000... [2024-09-05 09:23:34,132][00556] Num frames 8100... [2024-09-05 09:23:34,257][00556] Num frames 8200... [2024-09-05 09:23:34,382][00556] Num frames 8300... [2024-09-05 09:23:34,509][00556] Num frames 8400... [2024-09-05 09:23:34,648][00556] Avg episode rewards: #0: 24.704, true rewards: #0: 10.579 [2024-09-05 09:23:34,650][00556] Avg episode reward: 24.704, avg true_objective: 10.579 [2024-09-05 09:23:34,704][00556] Num frames 8500... [2024-09-05 09:23:34,828][00556] Num frames 8600... [2024-09-05 09:23:34,951][00556] Num frames 8700... [2024-09-05 09:23:35,083][00556] Num frames 8800... [2024-09-05 09:23:35,262][00556] Num frames 8900... [2024-09-05 09:23:35,440][00556] Num frames 9000... [2024-09-05 09:23:35,608][00556] Num frames 9100... [2024-09-05 09:23:35,786][00556] Num frames 9200... [2024-09-05 09:23:35,959][00556] Num frames 9300... [2024-09-05 09:23:36,131][00556] Num frames 9400... [2024-09-05 09:23:36,318][00556] Num frames 9500... [2024-09-05 09:23:36,493][00556] Num frames 9600... [2024-09-05 09:23:36,675][00556] Num frames 9700... [2024-09-05 09:23:36,804][00556] Avg episode rewards: #0: 25.270, true rewards: #0: 10.826 [2024-09-05 09:23:36,807][00556] Avg episode reward: 25.270, avg true_objective: 10.826 [2024-09-05 09:23:36,910][00556] Num frames 9800... [2024-09-05 09:23:37,091][00556] Num frames 9900... [2024-09-05 09:23:37,273][00556] Num frames 10000... [2024-09-05 09:23:37,455][00556] Num frames 10100... [2024-09-05 09:23:37,633][00556] Num frames 10200... [2024-09-05 09:23:37,777][00556] Num frames 10300... [2024-09-05 09:23:37,856][00556] Avg episode rewards: #0: 23.719, true rewards: #0: 10.319 [2024-09-05 09:23:37,858][00556] Avg episode reward: 23.719, avg true_objective: 10.319 [2024-09-05 09:24:46,467][00556] Replay video saved to /content/train_dir/default_experiment/replay.mp4! [2024-09-05 09:24:46,948][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:24:46,950][00556] Overriding arg 'num_workers' with value 1 passed from command line [2024-09-05 09:24:46,952][00556] Adding new argument 'no_render'=True that is not in the saved config file! [2024-09-05 09:24:46,954][00556] Adding new argument 'save_video'=True that is not in the saved config file! [2024-09-05 09:24:46,956][00556] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:24:46,958][00556] Adding new argument 'video_name'=None that is not in the saved config file! [2024-09-05 09:24:46,960][00556] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2024-09-05 09:24:46,961][00556] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2024-09-05 09:24:46,962][00556] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2024-09-05 09:24:46,964][00556] Adding new argument 'hf_repository'='neeldevenshah/rl_course_vizdoom_health_gathering_supreme' that is not in the saved config file! [2024-09-05 09:24:46,965][00556] Adding new argument 'policy_index'=0 that is not in the saved config file! [2024-09-05 09:24:46,966][00556] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2024-09-05 09:24:46,967][00556] Adding new argument 'train_script'=None that is not in the saved config file! [2024-09-05 09:24:46,968][00556] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2024-09-05 09:24:46,969][00556] Using frameskip 1 and render_action_repeat=4 for evaluation [2024-09-05 09:24:47,009][00556] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:24:47,012][00556] RunningMeanStd input shape: (1,) [2024-09-05 09:24:47,032][00556] ConvEncoder: input_channels=3 [2024-09-05 09:24:47,101][00556] Conv encoder output size: 512 [2024-09-05 09:24:47,103][00556] Policy head output size: 512 [2024-09-05 09:24:47,130][00556] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000981_4018176.pth... [2024-09-05 09:24:47,796][00556] Num frames 100... [2024-09-05 09:24:47,962][00556] Num frames 200... [2024-09-05 09:24:48,132][00556] Num frames 300... [2024-09-05 09:24:48,299][00556] Num frames 400... [2024-09-05 09:24:48,468][00556] Num frames 500... [2024-09-05 09:24:48,630][00556] Num frames 600... [2024-09-05 09:24:48,798][00556] Num frames 700... [2024-09-05 09:24:48,956][00556] Num frames 800... [2024-09-05 09:24:49,119][00556] Num frames 900... [2024-09-05 09:24:49,294][00556] Num frames 1000... [2024-09-05 09:24:49,465][00556] Num frames 1100... [2024-09-05 09:24:49,630][00556] Num frames 1200... [2024-09-05 09:24:49,798][00556] Num frames 1300... [2024-09-05 09:24:49,970][00556] Num frames 1400... [2024-09-05 09:24:50,147][00556] Num frames 1500... [2024-09-05 09:24:50,321][00556] Num frames 1600... [2024-09-05 09:24:50,516][00556] Num frames 1700... [2024-09-05 09:24:50,697][00556] Num frames 1800... [2024-09-05 09:24:50,811][00556] Avg episode rewards: #0: 42.329, true rewards: #0: 18.330 [2024-09-05 09:24:50,814][00556] Avg episode reward: 42.329, avg true_objective: 18.330 [2024-09-05 09:24:50,935][00556] Num frames 1900... [2024-09-05 09:24:51,117][00556] Num frames 2000... [2024-09-05 09:24:51,343][00556] Num frames 2100... [2024-09-05 09:24:51,564][00556] Num frames 2200... [2024-09-05 09:24:51,746][00556] Num frames 2300... [2024-09-05 09:24:51,930][00556] Num frames 2400... [2024-09-05 09:24:52,128][00556] Num frames 2500... [2024-09-05 09:24:52,359][00556] Num frames 2600... [2024-09-05 09:24:52,562][00556] Num frames 2700... [2024-09-05 09:24:52,746][00556] Num frames 2800... [2024-09-05 09:24:52,931][00556] Num frames 2900... [2024-09-05 09:24:53,107][00556] Num frames 3000... [2024-09-05 09:24:53,275][00556] Num frames 3100... [2024-09-05 09:24:53,457][00556] Num frames 3200... [2024-09-05 09:24:53,630][00556] Num frames 3300... [2024-09-05 09:24:53,798][00556] Num frames 3400... [2024-09-05 09:24:53,953][00556] Avg episode rewards: #0: 40.874, true rewards: #0: 17.375 [2024-09-05 09:24:53,955][00556] Avg episode reward: 40.874, avg true_objective: 17.375 [2024-09-05 09:24:53,989][00556] Num frames 3500... [2024-09-05 09:24:54,109][00556] Num frames 3600... [2024-09-05 09:24:54,234][00556] Num frames 3700... [2024-09-05 09:24:54,358][00556] Num frames 3800... [2024-09-05 09:24:54,491][00556] Num frames 3900... [2024-09-05 09:24:54,616][00556] Num frames 4000... [2024-09-05 09:24:54,748][00556] Num frames 4100... [2024-09-05 09:24:54,861][00556] Avg episode rewards: #0: 31.823, true rewards: #0: 13.823 [2024-09-05 09:24:54,862][00556] Avg episode reward: 31.823, avg true_objective: 13.823 [2024-09-05 09:24:54,964][00556] Num frames 4200... [2024-09-05 09:24:55,144][00556] Num frames 4300... [2024-09-05 09:24:55,309][00556] Num frames 4400... [2024-09-05 09:24:55,484][00556] Num frames 4500... [2024-09-05 09:24:55,671][00556] Num frames 4600... [2024-09-05 09:24:55,846][00556] Num frames 4700... [2024-09-05 09:24:56,011][00556] Num frames 4800... [2024-09-05 09:24:56,182][00556] Num frames 4900... [2024-09-05 09:24:56,267][00556] Avg episode rewards: #0: 27.287, true rewards: #0: 12.287 [2024-09-05 09:24:56,269][00556] Avg episode reward: 27.287, avg true_objective: 12.287 [2024-09-05 09:24:56,427][00556] Num frames 5000... [2024-09-05 09:24:56,605][00556] Num frames 5100... [2024-09-05 09:24:56,802][00556] Num frames 5200... [2024-09-05 09:24:56,984][00556] Num frames 5300... [2024-09-05 09:24:57,165][00556] Num frames 5400... [2024-09-05 09:24:57,342][00556] Num frames 5500... [2024-09-05 09:24:57,519][00556] Num frames 5600... [2024-09-05 09:24:57,599][00556] Avg episode rewards: #0: 24.438, true rewards: #0: 11.238 [2024-09-05 09:24:57,601][00556] Avg episode reward: 24.438, avg true_objective: 11.238 [2024-09-05 09:24:57,712][00556] Num frames 5700... [2024-09-05 09:24:57,836][00556] Num frames 5800... [2024-09-05 09:24:57,962][00556] Num frames 5900... [2024-09-05 09:24:58,088][00556] Num frames 6000... [2024-09-05 09:24:58,213][00556] Num frames 6100... [2024-09-05 09:24:58,358][00556] Avg episode rewards: #0: 22.288, true rewards: #0: 10.288 [2024-09-05 09:24:58,360][00556] Avg episode reward: 22.288, avg true_objective: 10.288 [2024-09-05 09:24:58,397][00556] Num frames 6200... [2024-09-05 09:24:58,525][00556] Num frames 6300... [2024-09-05 09:24:58,664][00556] Num frames 6400... [2024-09-05 09:24:58,791][00556] Num frames 6500... [2024-09-05 09:24:58,892][00556] Avg episode rewards: #0: 19.908, true rewards: #0: 9.337 [2024-09-05 09:24:58,893][00556] Avg episode reward: 19.908, avg true_objective: 9.337 [2024-09-05 09:24:58,975][00556] Num frames 6600... [2024-09-05 09:24:59,097][00556] Num frames 6700... [2024-09-05 09:24:59,219][00556] Num frames 6800... [2024-09-05 09:24:59,344][00556] Num frames 6900... [2024-09-05 09:24:59,471][00556] Num frames 7000... [2024-09-05 09:24:59,599][00556] Num frames 7100... [2024-09-05 09:24:59,675][00556] Avg episode rewards: #0: 18.640, true rewards: #0: 8.890 [2024-09-05 09:24:59,677][00556] Avg episode reward: 18.640, avg true_objective: 8.890 [2024-09-05 09:24:59,792][00556] Num frames 7200... [2024-09-05 09:24:59,914][00556] Num frames 7300... [2024-09-05 09:25:00,042][00556] Num frames 7400... [2024-09-05 09:25:00,176][00556] Num frames 7500... [2024-09-05 09:25:00,300][00556] Num frames 7600... [2024-09-05 09:25:00,424][00556] Num frames 7700... [2024-09-05 09:25:00,549][00556] Num frames 7800... [2024-09-05 09:25:00,686][00556] Num frames 7900... [2024-09-05 09:25:00,810][00556] Num frames 8000... [2024-09-05 09:25:00,890][00556] Avg episode rewards: #0: 18.467, true rewards: #0: 8.911 [2024-09-05 09:25:00,891][00556] Avg episode reward: 18.467, avg true_objective: 8.911 [2024-09-05 09:25:00,994][00556] Num frames 8100... [2024-09-05 09:25:01,118][00556] Num frames 8200... [2024-09-05 09:25:01,266][00556] Num frames 8300... [2024-09-05 09:25:01,391][00556] Num frames 8400... [2024-09-05 09:25:01,516][00556] Num frames 8500... [2024-09-05 09:25:01,641][00556] Num frames 8600... [2024-09-05 09:25:01,782][00556] Num frames 8700... [2024-09-05 09:25:01,919][00556] Num frames 8800... [2024-09-05 09:25:02,001][00556] Avg episode rewards: #0: 18.120, true rewards: #0: 8.820 [2024-09-05 09:25:02,002][00556] Avg episode reward: 18.120, avg true_objective: 8.820 [2024-09-05 09:25:58,542][00556] Replay video saved to /content/train_dir/default_experiment/replay.mp4! [2024-09-05 09:26:04,429][00556] The model has been pushed to https://huggingface.co./neeldevenshah/rl_course_vizdoom_health_gathering_supreme [2024-09-05 09:27:51,830][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:27:51,831][00556] Overriding arg 'train_for_env_steps' with value 8000000 passed from command line [2024-09-05 09:27:51,838][00556] Experiment dir /content/train_dir/default_experiment already exists! [2024-09-05 09:27:51,839][00556] Resuming existing experiment from /content/train_dir/default_experiment... [2024-09-05 09:27:51,840][00556] Weights and Biases integration disabled [2024-09-05 09:27:51,845][00556] Environment var CUDA_VISIBLE_DEVICES is 0 [2024-09-05 09:27:54,040][00556] Starting experiment with the following configuration: help=False algo=APPO env=doom_health_gathering_supreme experiment=default_experiment train_dir=/content/train_dir restart_behavior=resume device=gpu seed=None num_policies=1 async_rl=True serial_mode=False batched_sampling=False num_batches_to_accumulate=2 worker_num_splits=2 policy_workers_per_policy=1 max_policy_lag=1000 num_workers=8 num_envs_per_worker=4 batch_size=1024 num_batches_per_epoch=1 num_epochs=1 rollout=32 recurrence=32 shuffle_minibatches=False gamma=0.99 reward_scale=1.0 reward_clip=1000.0 value_bootstrap=False normalize_returns=True exploration_loss_coeff=0.001 value_loss_coeff=0.5 kl_loss_coeff=0.0 exploration_loss=symmetric_kl gae_lambda=0.95 ppo_clip_ratio=0.1 ppo_clip_value=0.2 with_vtrace=False vtrace_rho=1.0 vtrace_c=1.0 optimizer=adam adam_eps=1e-06 adam_beta1=0.9 adam_beta2=0.999 max_grad_norm=4.0 learning_rate=0.0001 lr_schedule=constant lr_schedule_kl_threshold=0.008 lr_adaptive_min=1e-06 lr_adaptive_max=0.01 obs_subtract_mean=0.0 obs_scale=255.0 normalize_input=True normalize_input_keys=None decorrelate_experience_max_seconds=0 decorrelate_envs_on_one_worker=True actor_worker_gpus=[] set_workers_cpu_affinity=True force_envs_single_thread=False default_niceness=0 log_to_file=True experiment_summaries_interval=10 flush_summaries_interval=30 stats_avg=100 summaries_use_frameskip=True heartbeat_interval=20 heartbeat_reporting_interval=600 train_for_env_steps=8000000 train_for_seconds=10000000000 save_every_sec=120 keep_checkpoints=2 load_checkpoint_kind=latest save_milestones_sec=-1 save_best_every_sec=5 save_best_metric=reward save_best_after=100000 benchmark=False encoder_mlp_layers=[512, 512] encoder_conv_architecture=convnet_simple encoder_conv_mlp_layers=[512] use_rnn=True rnn_size=512 rnn_type=gru rnn_num_layers=1 decoder_mlp_layers=[] nonlinearity=elu policy_initialization=orthogonal policy_init_gain=1.0 actor_critic_share_weights=True adaptive_stddev=True continuous_tanh_scale=0.0 initial_stddev=1.0 use_env_info_cache=False env_gpu_actions=False env_gpu_observations=True env_frameskip=4 env_framestack=1 pixel_format=CHW use_record_episode_statistics=False with_wandb=False wandb_user=None wandb_project=sample_factory wandb_group=None wandb_job_type=SF wandb_tags=[] with_pbt=False pbt_mix_policies_in_one_env=True pbt_period_env_steps=5000000 pbt_start_mutation=20000000 pbt_replace_fraction=0.3 pbt_mutation_rate=0.15 pbt_replace_reward_gap=0.1 pbt_replace_reward_gap_absolute=1e-06 pbt_optimize_gamma=False pbt_target_objective=true_objective pbt_perturb_min=1.1 pbt_perturb_max=1.5 num_agents=-1 num_humans=0 num_bots=-1 start_bot_difficulty=None timelimit=None res_w=128 res_h=72 wide_aspect_ratio=False eval_env_frameskip=1 fps=35 command_line=--env=doom_health_gathering_supreme --num_workers=8 --num_envs_per_worker=4 --train_for_env_steps=4000000 cli_args={'env': 'doom_health_gathering_supreme', 'num_workers': 8, 'num_envs_per_worker': 4, 'train_for_env_steps': 4000000} git_hash=unknown git_repo_name=not a git repository [2024-09-05 09:27:54,043][00556] Saving configuration to /content/train_dir/default_experiment/config.json... [2024-09-05 09:27:54,046][00556] Rollout worker 0 uses device cpu [2024-09-05 09:27:54,049][00556] Rollout worker 1 uses device cpu [2024-09-05 09:27:54,050][00556] Rollout worker 2 uses device cpu [2024-09-05 09:27:54,051][00556] Rollout worker 3 uses device cpu [2024-09-05 09:27:54,053][00556] Rollout worker 4 uses device cpu [2024-09-05 09:27:54,054][00556] Rollout worker 5 uses device cpu [2024-09-05 09:27:54,055][00556] Rollout worker 6 uses device cpu [2024-09-05 09:27:54,056][00556] Rollout worker 7 uses device cpu [2024-09-05 09:27:54,133][00556] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:27:54,134][00556] InferenceWorker_p0-w0: min num requests: 2 [2024-09-05 09:27:54,166][00556] Starting all processes... [2024-09-05 09:27:54,168][00556] Starting process learner_proc0 [2024-09-05 09:27:54,216][00556] Starting all processes... [2024-09-05 09:27:54,223][00556] Starting process inference_proc0-0 [2024-09-05 09:27:54,224][00556] Starting process rollout_proc0 [2024-09-05 09:27:54,228][00556] Starting process rollout_proc1 [2024-09-05 09:27:54,248][00556] Starting process rollout_proc2 [2024-09-05 09:27:54,249][00556] Starting process rollout_proc3 [2024-09-05 09:27:54,249][00556] Starting process rollout_proc4 [2024-09-05 09:27:54,249][00556] Starting process rollout_proc5 [2024-09-05 09:27:54,251][00556] Starting process rollout_proc6 [2024-09-05 09:27:54,251][00556] Starting process rollout_proc7 [2024-09-05 09:28:10,499][15082] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:28:10,509][15082] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 [2024-09-05 09:28:10,581][15082] Num visible devices: 1 [2024-09-05 09:28:10,620][15082] Starting seed is not provided [2024-09-05 09:28:10,621][15082] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:28:10,622][15082] Initializing actor-critic model on device cuda:0 [2024-09-05 09:28:10,623][15082] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:28:10,625][15082] RunningMeanStd input shape: (1,) [2024-09-05 09:28:10,780][15082] ConvEncoder: input_channels=3 [2024-09-05 09:28:10,961][15101] Worker 5 uses CPU cores [1] [2024-09-05 09:28:11,044][15100] Worker 2 uses CPU cores [0] [2024-09-05 09:28:11,196][15103] Worker 7 uses CPU cores [1] [2024-09-05 09:28:11,217][15095] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:28:11,217][15095] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 [2024-09-05 09:28:11,266][15098] Worker 3 uses CPU cores [1] [2024-09-05 09:28:11,314][15095] Num visible devices: 1 [2024-09-05 09:28:11,418][15096] Worker 0 uses CPU cores [0] [2024-09-05 09:28:11,495][15097] Worker 1 uses CPU cores [1] [2024-09-05 09:28:11,508][15099] Worker 4 uses CPU cores [0] [2024-09-05 09:28:11,514][15102] Worker 6 uses CPU cores [0] [2024-09-05 09:28:11,593][15082] Conv encoder output size: 512 [2024-09-05 09:28:11,594][15082] Policy head output size: 512 [2024-09-05 09:28:11,623][15082] Created Actor Critic model with architecture: [2024-09-05 09:28:11,624][15082] ActorCriticSharedWeights( (obs_normalizer): ObservationNormalizer( (running_mean_std): RunningMeanStdDictInPlace( (running_mean_std): ModuleDict( (obs): RunningMeanStdInPlace() ) ) ) (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) (encoder): VizdoomEncoder( (basic_encoder): ConvEncoder( (enc): RecursiveScriptModule( original_name=ConvEncoderImpl (conv_head): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Conv2d) (1): RecursiveScriptModule(original_name=ELU) (2): RecursiveScriptModule(original_name=Conv2d) (3): RecursiveScriptModule(original_name=ELU) (4): RecursiveScriptModule(original_name=Conv2d) (5): RecursiveScriptModule(original_name=ELU) ) (mlp_layers): RecursiveScriptModule( original_name=Sequential (0): RecursiveScriptModule(original_name=Linear) (1): RecursiveScriptModule(original_name=ELU) ) ) ) ) (core): ModelCoreRNN( (core): GRU(512, 512) ) (decoder): MlpDecoder( (mlp): Identity() ) (critic_linear): Linear(in_features=512, out_features=1, bias=True) (action_parameterization): ActionParameterizationDefault( (distribution_linear): Linear(in_features=512, out_features=5, bias=True) ) ) [2024-09-05 09:28:11,796][15082] Using optimizer [2024-09-05 09:28:12,678][15082] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000981_4018176.pth... [2024-09-05 09:28:12,732][15082] Loading model from checkpoint [2024-09-05 09:28:12,735][15082] Loaded experiment state at self.train_step=981, self.env_steps=4018176 [2024-09-05 09:28:12,735][15082] Initialized policy 0 weights for model version 981 [2024-09-05 09:28:12,746][15082] Using GPUs [0] for process 0 (actually maps to GPUs [0]) [2024-09-05 09:28:12,755][15082] LearnerWorker_p0 finished initialization! [2024-09-05 09:28:12,989][15095] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:28:12,991][15095] RunningMeanStd input shape: (1,) [2024-09-05 09:28:13,140][15095] ConvEncoder: input_channels=3 [2024-09-05 09:28:13,412][15095] Conv encoder output size: 512 [2024-09-05 09:28:13,413][15095] Policy head output size: 512 [2024-09-05 09:28:13,498][00556] Inference worker 0-0 is ready! [2024-09-05 09:28:13,500][00556] All inference workers are ready! Signal rollout workers to start! [2024-09-05 09:28:13,914][15096] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:13,888][15099] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:13,931][15102] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:13,959][15103] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:13,982][15098] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:13,898][15097] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:13,994][15101] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:14,032][15100] Doom resolution: 160x120, resize resolution: (128, 72) [2024-09-05 09:28:14,125][00556] Heartbeat connected on Batcher_0 [2024-09-05 09:28:14,134][00556] Heartbeat connected on LearnerWorker_p0 [2024-09-05 09:28:14,182][00556] Heartbeat connected on InferenceWorker_p0-w0 [2024-09-05 09:28:16,113][15102] Decorrelating experience for 0 frames... [2024-09-05 09:28:16,121][15099] Decorrelating experience for 0 frames... [2024-09-05 09:28:16,119][15096] Decorrelating experience for 0 frames... [2024-09-05 09:28:16,120][15101] Decorrelating experience for 0 frames... [2024-09-05 09:28:16,125][15103] Decorrelating experience for 0 frames... [2024-09-05 09:28:16,126][15097] Decorrelating experience for 0 frames... [2024-09-05 09:28:16,845][00556] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 4018176. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 09:28:17,322][15103] Decorrelating experience for 32 frames... [2024-09-05 09:28:17,328][15101] Decorrelating experience for 32 frames... [2024-09-05 09:28:17,392][15098] Decorrelating experience for 0 frames... [2024-09-05 09:28:17,658][15099] Decorrelating experience for 32 frames... [2024-09-05 09:28:17,671][15102] Decorrelating experience for 32 frames... [2024-09-05 09:28:17,676][15096] Decorrelating experience for 32 frames... [2024-09-05 09:28:17,748][15100] Decorrelating experience for 0 frames... [2024-09-05 09:28:18,467][15102] Decorrelating experience for 64 frames... [2024-09-05 09:28:18,702][15103] Decorrelating experience for 64 frames... [2024-09-05 09:28:18,705][15101] Decorrelating experience for 64 frames... [2024-09-05 09:28:18,884][15097] Decorrelating experience for 32 frames... [2024-09-05 09:28:19,296][15102] Decorrelating experience for 96 frames... [2024-09-05 09:28:19,490][00556] Heartbeat connected on RolloutWorker_w6 [2024-09-05 09:28:20,257][15099] Decorrelating experience for 64 frames... [2024-09-05 09:28:20,274][15100] Decorrelating experience for 32 frames... [2024-09-05 09:28:20,598][15103] Decorrelating experience for 96 frames... [2024-09-05 09:28:20,613][15101] Decorrelating experience for 96 frames... [2024-09-05 09:28:20,692][15098] Decorrelating experience for 32 frames... [2024-09-05 09:28:20,948][00556] Heartbeat connected on RolloutWorker_w5 [2024-09-05 09:28:20,953][00556] Heartbeat connected on RolloutWorker_w7 [2024-09-05 09:28:21,199][15097] Decorrelating experience for 64 frames... [2024-09-05 09:28:21,845][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4018176. Throughput: 0: 2.4. Samples: 12. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 09:28:21,852][00556] Avg episode reward: [(0, '1.280')] [2024-09-05 09:28:22,446][15096] Decorrelating experience for 64 frames... [2024-09-05 09:28:22,587][15099] Decorrelating experience for 96 frames... [2024-09-05 09:28:22,859][00556] Heartbeat connected on RolloutWorker_w4 [2024-09-05 09:28:23,035][15100] Decorrelating experience for 64 frames... [2024-09-05 09:28:25,744][15098] Decorrelating experience for 64 frames... [2024-09-05 09:28:26,845][00556] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 4018176. Throughput: 0: 199.6. Samples: 1996. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) [2024-09-05 09:28:26,848][00556] Avg episode reward: [(0, '6.100')] [2024-09-05 09:28:27,105][15082] Signal inference workers to stop experience collection... [2024-09-05 09:28:27,145][15095] InferenceWorker_p0-w0: stopping experience collection [2024-09-05 09:28:27,966][15096] Decorrelating experience for 96 frames... [2024-09-05 09:28:28,785][00556] Heartbeat connected on RolloutWorker_w0 [2024-09-05 09:28:29,293][15082] Signal inference workers to resume experience collection... [2024-09-05 09:28:29,294][15095] InferenceWorker_p0-w0: resuming experience collection [2024-09-05 09:28:29,583][15100] Decorrelating experience for 96 frames... [2024-09-05 09:28:30,851][00556] Heartbeat connected on RolloutWorker_w2 [2024-09-05 09:28:31,751][15097] Decorrelating experience for 96 frames... [2024-09-05 09:28:31,845][00556] Fps is (10 sec: 819.2, 60 sec: 546.1, 300 sec: 546.1). Total num frames: 4026368. Throughput: 0: 198.5. Samples: 2978. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) [2024-09-05 09:28:31,848][00556] Avg episode reward: [(0, '5.377')] [2024-09-05 09:28:32,295][15098] Decorrelating experience for 96 frames... [2024-09-05 09:28:32,678][00556] Heartbeat connected on RolloutWorker_w1 [2024-09-05 09:28:33,251][00556] Heartbeat connected on RolloutWorker_w3 [2024-09-05 09:28:36,846][00556] Fps is (10 sec: 2047.8, 60 sec: 1023.9, 300 sec: 1023.9). Total num frames: 4038656. Throughput: 0: 220.6. Samples: 4412. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:28:36,855][00556] Avg episode reward: [(0, '9.302')] [2024-09-05 09:28:41,305][15095] Updated weights for policy 0, policy_version 991 (0.0034) [2024-09-05 09:28:41,845][00556] Fps is (10 sec: 3276.8, 60 sec: 1638.4, 300 sec: 1638.4). Total num frames: 4059136. Throughput: 0: 395.1. Samples: 9878. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:28:41,848][00556] Avg episode reward: [(0, '10.941')] [2024-09-05 09:28:46,845][00556] Fps is (10 sec: 4506.1, 60 sec: 2184.5, 300 sec: 2184.5). Total num frames: 4083712. Throughput: 0: 555.1. Samples: 16654. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:28:46,850][00556] Avg episode reward: [(0, '14.391')] [2024-09-05 09:28:51,845][00556] Fps is (10 sec: 3686.4, 60 sec: 2223.5, 300 sec: 2223.5). Total num frames: 4096000. Throughput: 0: 539.0. Samples: 18864. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:28:51,849][00556] Avg episode reward: [(0, '16.882')] [2024-09-05 09:28:52,739][15095] Updated weights for policy 0, policy_version 1001 (0.0022) [2024-09-05 09:28:56,845][00556] Fps is (10 sec: 2867.2, 60 sec: 2355.2, 300 sec: 2355.2). Total num frames: 4112384. Throughput: 0: 583.6. Samples: 23344. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:28:56,852][00556] Avg episode reward: [(0, '18.192')] [2024-09-05 09:29:01,845][00556] Fps is (10 sec: 4095.9, 60 sec: 2639.6, 300 sec: 2639.6). Total num frames: 4136960. Throughput: 0: 670.4. Samples: 30168. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:29:01,847][00556] Avg episode reward: [(0, '21.429')] [2024-09-05 09:29:02,576][15095] Updated weights for policy 0, policy_version 1011 (0.0029) [2024-09-05 09:29:06,845][00556] Fps is (10 sec: 4096.0, 60 sec: 2703.4, 300 sec: 2703.4). Total num frames: 4153344. Throughput: 0: 746.0. Samples: 33582. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:29:06,852][00556] Avg episode reward: [(0, '22.022')] [2024-09-05 09:29:11,845][00556] Fps is (10 sec: 3276.9, 60 sec: 2755.5, 300 sec: 2755.5). Total num frames: 4169728. Throughput: 0: 795.6. Samples: 37796. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:29:11,847][00556] Avg episode reward: [(0, '24.475')] [2024-09-05 09:29:11,864][15082] Saving new best policy, reward=24.475! [2024-09-05 09:29:14,541][15095] Updated weights for policy 0, policy_version 1021 (0.0026) [2024-09-05 09:29:16,845][00556] Fps is (10 sec: 3686.4, 60 sec: 2867.2, 300 sec: 2867.2). Total num frames: 4190208. Throughput: 0: 908.7. Samples: 43870. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:29:16,848][00556] Avg episode reward: [(0, '22.554')] [2024-09-05 09:29:21,845][00556] Fps is (10 sec: 4505.5, 60 sec: 3276.8, 300 sec: 3024.7). Total num frames: 4214784. Throughput: 0: 950.9. Samples: 47202. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:29:21,848][00556] Avg episode reward: [(0, '21.062')] [2024-09-05 09:29:23,862][15095] Updated weights for policy 0, policy_version 1031 (0.0015) [2024-09-05 09:29:26,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3481.6, 300 sec: 2984.2). Total num frames: 4227072. Throughput: 0: 952.4. Samples: 52734. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:29:26,861][00556] Avg episode reward: [(0, '20.565')] [2024-09-05 09:29:31,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3058.3). Total num frames: 4247552. Throughput: 0: 912.1. Samples: 57698. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:29:31,847][00556] Avg episode reward: [(0, '20.991')] [2024-09-05 09:29:35,102][15095] Updated weights for policy 0, policy_version 1041 (0.0052) [2024-09-05 09:29:36,846][00556] Fps is (10 sec: 4095.6, 60 sec: 3822.9, 300 sec: 3123.2). Total num frames: 4268032. Throughput: 0: 938.8. Samples: 61110. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:29:36,854][00556] Avg episode reward: [(0, '22.126')] [2024-09-05 09:29:41,845][00556] Fps is (10 sec: 4096.1, 60 sec: 3822.9, 300 sec: 3180.4). Total num frames: 4288512. Throughput: 0: 984.8. Samples: 67658. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:29:41,850][00556] Avg episode reward: [(0, '21.720')] [2024-09-05 09:29:46,845][00556] Fps is (10 sec: 3277.1, 60 sec: 3618.1, 300 sec: 3140.3). Total num frames: 4300800. Throughput: 0: 927.8. Samples: 71918. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:29:46,852][00556] Avg episode reward: [(0, '22.858')] [2024-09-05 09:29:46,987][15095] Updated weights for policy 0, policy_version 1051 (0.0022) [2024-09-05 09:29:51,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3233.7). Total num frames: 4325376. Throughput: 0: 918.6. Samples: 74918. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:29:51,850][00556] Avg episode reward: [(0, '22.899')] [2024-09-05 09:29:51,860][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001056_4325376.pth... [2024-09-05 09:29:51,987][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000980_4014080.pth [2024-09-05 09:29:56,185][15095] Updated weights for policy 0, policy_version 1061 (0.0030) [2024-09-05 09:29:56,848][00556] Fps is (10 sec: 4504.4, 60 sec: 3891.0, 300 sec: 3276.7). Total num frames: 4345856. Throughput: 0: 975.8. Samples: 81710. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:29:56,853][00556] Avg episode reward: [(0, '23.362')] [2024-09-05 09:30:01,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3276.8). Total num frames: 4362240. Throughput: 0: 954.2. Samples: 86808. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:30:01,854][00556] Avg episode reward: [(0, '22.478')] [2024-09-05 09:30:06,845][00556] Fps is (10 sec: 3277.7, 60 sec: 3754.7, 300 sec: 3276.8). Total num frames: 4378624. Throughput: 0: 926.3. Samples: 88884. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:30:06,850][00556] Avg episode reward: [(0, '22.636')] [2024-09-05 09:30:08,110][15095] Updated weights for policy 0, policy_version 1071 (0.0017) [2024-09-05 09:30:11,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3348.0). Total num frames: 4403200. Throughput: 0: 948.1. Samples: 95398. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:30:11,851][00556] Avg episode reward: [(0, '23.126')] [2024-09-05 09:30:16,846][00556] Fps is (10 sec: 4095.6, 60 sec: 3822.9, 300 sec: 3345.0). Total num frames: 4419584. Throughput: 0: 977.4. Samples: 101682. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:30:16,855][00556] Avg episode reward: [(0, '22.725')] [2024-09-05 09:30:18,509][15095] Updated weights for policy 0, policy_version 1081 (0.0031) [2024-09-05 09:30:21,845][00556] Fps is (10 sec: 3276.9, 60 sec: 3686.4, 300 sec: 3342.3). Total num frames: 4435968. Throughput: 0: 946.1. Samples: 103684. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:30:21,851][00556] Avg episode reward: [(0, '22.778')] [2024-09-05 09:30:26,845][00556] Fps is (10 sec: 3686.8, 60 sec: 3822.9, 300 sec: 3371.3). Total num frames: 4456448. Throughput: 0: 923.2. Samples: 109204. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:30:26,852][00556] Avg episode reward: [(0, '21.538')] [2024-09-05 09:30:29,118][15095] Updated weights for policy 0, policy_version 1091 (0.0026) [2024-09-05 09:30:31,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3428.5). Total num frames: 4481024. Throughput: 0: 984.5. Samples: 116220. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:30:31,852][00556] Avg episode reward: [(0, '21.585')] [2024-09-05 09:30:36,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3823.0, 300 sec: 3423.1). Total num frames: 4497408. Throughput: 0: 975.6. Samples: 118822. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:30:36,849][00556] Avg episode reward: [(0, '21.406')] [2024-09-05 09:30:40,892][15095] Updated weights for policy 0, policy_version 1101 (0.0028) [2024-09-05 09:30:41,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3418.0). Total num frames: 4513792. Throughput: 0: 916.2. Samples: 122938. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:30:41,851][00556] Avg episode reward: [(0, '22.522')] [2024-09-05 09:30:46,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3440.6). Total num frames: 4534272. Throughput: 0: 957.8. Samples: 129910. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:30:46,847][00556] Avg episode reward: [(0, '21.149')] [2024-09-05 09:30:49,860][15095] Updated weights for policy 0, policy_version 1111 (0.0032) [2024-09-05 09:30:51,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3461.8). Total num frames: 4554752. Throughput: 0: 989.0. Samples: 133390. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:30:51,847][00556] Avg episode reward: [(0, '21.726')] [2024-09-05 09:30:56,848][00556] Fps is (10 sec: 3685.3, 60 sec: 3754.6, 300 sec: 3455.9). Total num frames: 4571136. Throughput: 0: 947.9. Samples: 138058. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:30:56,852][00556] Avg episode reward: [(0, '21.112')] [2024-09-05 09:31:01,517][15095] Updated weights for policy 0, policy_version 1121 (0.0026) [2024-09-05 09:31:01,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3475.4). Total num frames: 4591616. Throughput: 0: 937.2. Samples: 143856. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:31:01,850][00556] Avg episode reward: [(0, '20.799')] [2024-09-05 09:31:06,845][00556] Fps is (10 sec: 4097.3, 60 sec: 3891.2, 300 sec: 3493.6). Total num frames: 4612096. Throughput: 0: 969.0. Samples: 147288. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:31:06,847][00556] Avg episode reward: [(0, '20.544')] [2024-09-05 09:31:11,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3487.4). Total num frames: 4628480. Throughput: 0: 974.1. Samples: 153038. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:31:11,851][00556] Avg episode reward: [(0, '22.286')] [2024-09-05 09:31:12,181][15095] Updated weights for policy 0, policy_version 1131 (0.0022) [2024-09-05 09:31:16,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3481.6). Total num frames: 4644864. Throughput: 0: 921.1. Samples: 157670. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:31:16,850][00556] Avg episode reward: [(0, '23.865')] [2024-09-05 09:31:21,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3520.3). Total num frames: 4669440. Throughput: 0: 941.6. Samples: 161192. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:31:21,852][00556] Avg episode reward: [(0, '25.017')] [2024-09-05 09:31:21,863][15082] Saving new best policy, reward=25.017! [2024-09-05 09:31:22,398][15095] Updated weights for policy 0, policy_version 1141 (0.0024) [2024-09-05 09:31:26,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3535.5). Total num frames: 4689920. Throughput: 0: 1002.4. Samples: 168048. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:31:26,851][00556] Avg episode reward: [(0, '25.733')] [2024-09-05 09:31:26,855][15082] Saving new best policy, reward=25.733! [2024-09-05 09:31:31,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3528.9). Total num frames: 4706304. Throughput: 0: 940.9. Samples: 172252. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:31:31,847][00556] Avg episode reward: [(0, '26.322')] [2024-09-05 09:31:31,865][15082] Saving new best policy, reward=26.322! [2024-09-05 09:31:34,124][15095] Updated weights for policy 0, policy_version 1151 (0.0044) [2024-09-05 09:31:36,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3543.0). Total num frames: 4726784. Throughput: 0: 922.5. Samples: 174904. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:31:36,850][00556] Avg episode reward: [(0, '26.651')] [2024-09-05 09:31:36,853][15082] Saving new best policy, reward=26.651! [2024-09-05 09:31:41,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3556.5). Total num frames: 4747264. Throughput: 0: 966.7. Samples: 181556. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:31:41,848][00556] Avg episode reward: [(0, '24.023')] [2024-09-05 09:31:43,480][15095] Updated weights for policy 0, policy_version 1161 (0.0016) [2024-09-05 09:31:46,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3549.9). Total num frames: 4763648. Throughput: 0: 955.6. Samples: 186860. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:31:46,848][00556] Avg episode reward: [(0, '23.279')] [2024-09-05 09:31:51,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3543.5). Total num frames: 4780032. Throughput: 0: 926.7. Samples: 188988. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:31:51,847][00556] Avg episode reward: [(0, '22.193')] [2024-09-05 09:31:51,859][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001167_4780032.pth... [2024-09-05 09:31:51,985][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000981_4018176.pth [2024-09-05 09:31:55,419][15095] Updated weights for policy 0, policy_version 1171 (0.0028) [2024-09-05 09:31:56,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.1, 300 sec: 3556.1). Total num frames: 4800512. Throughput: 0: 937.3. Samples: 195218. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:31:56,847][00556] Avg episode reward: [(0, '23.028')] [2024-09-05 09:32:01,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3568.1). Total num frames: 4820992. Throughput: 0: 980.0. Samples: 201768. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:32:01,849][00556] Avg episode reward: [(0, '22.446')] [2024-09-05 09:32:06,751][15095] Updated weights for policy 0, policy_version 1181 (0.0024) [2024-09-05 09:32:06,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3561.7). Total num frames: 4837376. Throughput: 0: 946.4. Samples: 203778. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:32:06,852][00556] Avg episode reward: [(0, '21.130')] [2024-09-05 09:32:11,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3555.7). Total num frames: 4853760. Throughput: 0: 903.0. Samples: 208684. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:32:11,847][00556] Avg episode reward: [(0, '21.423')] [2024-09-05 09:32:16,420][15095] Updated weights for policy 0, policy_version 1191 (0.0038) [2024-09-05 09:32:16,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3584.0). Total num frames: 4878336. Throughput: 0: 964.3. Samples: 215644. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:32:16,847][00556] Avg episode reward: [(0, '22.584')] [2024-09-05 09:32:21,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3577.7). Total num frames: 4894720. Throughput: 0: 971.5. Samples: 218620. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:32:21,852][00556] Avg episode reward: [(0, '22.038')] [2024-09-05 09:32:26,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3571.7). Total num frames: 4911104. Throughput: 0: 917.7. Samples: 222854. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:32:26,850][00556] Avg episode reward: [(0, '21.314')] [2024-09-05 09:32:28,347][15095] Updated weights for policy 0, policy_version 1201 (0.0035) [2024-09-05 09:32:31,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3565.9). Total num frames: 4927488. Throughput: 0: 915.7. Samples: 228066. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:32:31,850][00556] Avg episode reward: [(0, '21.059')] [2024-09-05 09:32:36,845][00556] Fps is (10 sec: 2867.2, 60 sec: 3549.9, 300 sec: 3544.6). Total num frames: 4939776. Throughput: 0: 914.8. Samples: 230154. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:32:36,850][00556] Avg episode reward: [(0, '22.290')] [2024-09-05 09:32:41,845][00556] Fps is (10 sec: 2867.2, 60 sec: 3481.6, 300 sec: 3539.6). Total num frames: 4956160. Throughput: 0: 869.8. Samples: 234360. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:32:41,851][00556] Avg episode reward: [(0, '23.813')] [2024-09-05 09:32:42,951][15095] Updated weights for policy 0, policy_version 1211 (0.0033) [2024-09-05 09:32:46,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3549.9, 300 sec: 3549.9). Total num frames: 4976640. Throughput: 0: 847.8. Samples: 239918. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:32:46,853][00556] Avg episode reward: [(0, '23.624')] [2024-09-05 09:32:51,845][00556] Fps is (10 sec: 4095.9, 60 sec: 3618.1, 300 sec: 3559.8). Total num frames: 4997120. Throughput: 0: 879.4. Samples: 243350. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:32:51,853][00556] Avg episode reward: [(0, '23.383')] [2024-09-05 09:32:52,025][15095] Updated weights for policy 0, policy_version 1221 (0.0027) [2024-09-05 09:32:56,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3618.1, 300 sec: 3569.4). Total num frames: 5017600. Throughput: 0: 906.1. Samples: 249458. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:32:56,849][00556] Avg episode reward: [(0, '22.860')] [2024-09-05 09:33:01,845][00556] Fps is (10 sec: 3276.7, 60 sec: 3481.6, 300 sec: 3549.9). Total num frames: 5029888. Throughput: 0: 849.7. Samples: 253882. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:33:01,848][00556] Avg episode reward: [(0, '23.334')] [2024-09-05 09:33:03,816][15095] Updated weights for policy 0, policy_version 1231 (0.0027) [2024-09-05 09:33:06,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3618.1, 300 sec: 3573.4). Total num frames: 5054464. Throughput: 0: 859.9. Samples: 257316. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:33:06,847][00556] Avg episode reward: [(0, '22.053')] [2024-09-05 09:33:11,845][00556] Fps is (10 sec: 4505.7, 60 sec: 3686.4, 300 sec: 3582.3). Total num frames: 5074944. Throughput: 0: 916.6. Samples: 264100. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:33:11,850][00556] Avg episode reward: [(0, '21.364')] [2024-09-05 09:33:13,924][15095] Updated weights for policy 0, policy_version 1241 (0.0031) [2024-09-05 09:33:16,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3481.6, 300 sec: 3623.9). Total num frames: 5087232. Throughput: 0: 903.4. Samples: 268720. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2024-09-05 09:33:16,854][00556] Avg episode reward: [(0, '20.723')] [2024-09-05 09:33:21,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3693.3). Total num frames: 5107712. Throughput: 0: 908.3. Samples: 271028. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:33:21,852][00556] Avg episode reward: [(0, '21.677')] [2024-09-05 09:33:24,848][15095] Updated weights for policy 0, policy_version 1251 (0.0031) [2024-09-05 09:33:26,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 5132288. Throughput: 0: 969.2. Samples: 277974. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:33:26,851][00556] Avg episode reward: [(0, '23.140')] [2024-09-05 09:33:31,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3762.8). Total num frames: 5148672. Throughput: 0: 975.6. Samples: 283822. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:33:31,849][00556] Avg episode reward: [(0, '22.439')] [2024-09-05 09:33:36,817][15095] Updated weights for policy 0, policy_version 1261 (0.0019) [2024-09-05 09:33:36,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 5165056. Throughput: 0: 944.6. Samples: 285858. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:33:36,855][00556] Avg episode reward: [(0, '23.419')] [2024-09-05 09:33:41,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3735.0). Total num frames: 5185536. Throughput: 0: 938.0. Samples: 291666. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:33:41,848][00556] Avg episode reward: [(0, '22.524')] [2024-09-05 09:33:45,737][15095] Updated weights for policy 0, policy_version 1271 (0.0034) [2024-09-05 09:33:46,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 5210112. Throughput: 0: 993.0. Samples: 298568. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:33:46,851][00556] Avg episode reward: [(0, '22.333')] [2024-09-05 09:33:51,845][00556] Fps is (10 sec: 3686.3, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 5222400. Throughput: 0: 965.4. Samples: 300758. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:33:51,848][00556] Avg episode reward: [(0, '21.998')] [2024-09-05 09:33:51,944][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001276_5226496.pth... [2024-09-05 09:33:52,166][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001056_4325376.pth [2024-09-05 09:33:56,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 5242880. Throughput: 0: 917.6. Samples: 305392. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:33:56,853][00556] Avg episode reward: [(0, '22.549')] [2024-09-05 09:33:57,552][15095] Updated weights for policy 0, policy_version 1281 (0.0048) [2024-09-05 09:34:01,845][00556] Fps is (10 sec: 4096.1, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 5263360. Throughput: 0: 971.0. Samples: 312414. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:34:01,852][00556] Avg episode reward: [(0, '22.893')] [2024-09-05 09:34:06,846][00556] Fps is (10 sec: 4095.5, 60 sec: 3822.9, 300 sec: 3776.6). Total num frames: 5283840. Throughput: 0: 995.0. Samples: 315802. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:34:06,849][00556] Avg episode reward: [(0, '23.235')] [2024-09-05 09:34:07,491][15095] Updated weights for policy 0, policy_version 1291 (0.0038) [2024-09-05 09:34:11,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 5296128. Throughput: 0: 933.2. Samples: 319968. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:34:11,848][00556] Avg episode reward: [(0, '23.700')] [2024-09-05 09:34:16,845][00556] Fps is (10 sec: 3686.8, 60 sec: 3891.2, 300 sec: 3748.9). Total num frames: 5320704. Throughput: 0: 941.7. Samples: 326200. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:34:16,849][00556] Avg episode reward: [(0, '25.317')] [2024-09-05 09:34:18,311][15095] Updated weights for policy 0, policy_version 1301 (0.0030) [2024-09-05 09:34:21,845][00556] Fps is (10 sec: 4915.2, 60 sec: 3959.5, 300 sec: 3790.5). Total num frames: 5345280. Throughput: 0: 973.2. Samples: 329654. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:34:21,847][00556] Avg episode reward: [(0, '25.854')] [2024-09-05 09:34:26,848][00556] Fps is (10 sec: 3685.2, 60 sec: 3754.5, 300 sec: 3762.7). Total num frames: 5357568. Throughput: 0: 964.0. Samples: 335050. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:34:26,850][00556] Avg episode reward: [(0, '25.711')] [2024-09-05 09:34:30,099][15095] Updated weights for policy 0, policy_version 1311 (0.0031) [2024-09-05 09:34:31,845][00556] Fps is (10 sec: 2867.2, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 5373952. Throughput: 0: 925.9. Samples: 340232. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:34:31,847][00556] Avg episode reward: [(0, '26.315')] [2024-09-05 09:34:36,845][00556] Fps is (10 sec: 4097.3, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 5398528. Throughput: 0: 952.1. Samples: 343604. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:34:36,847][00556] Avg episode reward: [(0, '26.038')] [2024-09-05 09:34:38,735][15095] Updated weights for policy 0, policy_version 1321 (0.0024) [2024-09-05 09:34:41,848][00556] Fps is (10 sec: 4504.2, 60 sec: 3891.0, 300 sec: 3790.5). Total num frames: 5419008. Throughput: 0: 994.4. Samples: 350142. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:34:41,851][00556] Avg episode reward: [(0, '25.790')] [2024-09-05 09:34:46,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 5431296. Throughput: 0: 928.4. Samples: 354194. Policy #0 lag: (min: 0.0, avg: 0.2, max: 1.0) [2024-09-05 09:34:46,851][00556] Avg episode reward: [(0, '25.458')] [2024-09-05 09:34:51,053][15095] Updated weights for policy 0, policy_version 1331 (0.0022) [2024-09-05 09:34:51,845][00556] Fps is (10 sec: 3277.8, 60 sec: 3822.9, 300 sec: 3748.9). Total num frames: 5451776. Throughput: 0: 917.6. Samples: 357092. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:34:51,847][00556] Avg episode reward: [(0, '24.572')] [2024-09-05 09:34:56,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 5476352. Throughput: 0: 977.2. Samples: 363944. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:34:56,852][00556] Avg episode reward: [(0, '25.046')] [2024-09-05 09:35:01,794][15095] Updated weights for policy 0, policy_version 1341 (0.0038) [2024-09-05 09:35:01,847][00556] Fps is (10 sec: 4095.1, 60 sec: 3822.8, 300 sec: 3776.6). Total num frames: 5492736. Throughput: 0: 952.3. Samples: 369056. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:35:01,850][00556] Avg episode reward: [(0, '25.079')] [2024-09-05 09:35:06,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 5509120. Throughput: 0: 922.4. Samples: 371160. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:35:06,847][00556] Avg episode reward: [(0, '23.864')] [2024-09-05 09:35:11,845][00556] Fps is (10 sec: 3687.2, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 5529600. Throughput: 0: 950.6. Samples: 377824. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:35:11,852][00556] Avg episode reward: [(0, '23.044')] [2024-09-05 09:35:12,061][15095] Updated weights for policy 0, policy_version 1351 (0.0026) [2024-09-05 09:35:16,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 5550080. Throughput: 0: 973.2. Samples: 384026. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:35:16,849][00556] Avg episode reward: [(0, '21.591')] [2024-09-05 09:35:21,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3686.4, 300 sec: 3762.8). Total num frames: 5566464. Throughput: 0: 944.4. Samples: 386102. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:35:21,848][00556] Avg episode reward: [(0, '21.257')] [2024-09-05 09:35:23,847][15095] Updated weights for policy 0, policy_version 1361 (0.0038) [2024-09-05 09:35:26,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.1, 300 sec: 3748.9). Total num frames: 5586944. Throughput: 0: 921.6. Samples: 391610. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:35:26,847][00556] Avg episode reward: [(0, '20.007')] [2024-09-05 09:35:31,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 5607424. Throughput: 0: 985.5. Samples: 398540. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:35:31,847][00556] Avg episode reward: [(0, '20.458')] [2024-09-05 09:35:32,828][15095] Updated weights for policy 0, policy_version 1371 (0.0027) [2024-09-05 09:35:36,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 5623808. Throughput: 0: 979.7. Samples: 401178. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:35:36,850][00556] Avg episode reward: [(0, '21.781')] [2024-09-05 09:35:41,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.6, 300 sec: 3748.9). Total num frames: 5640192. Throughput: 0: 921.6. Samples: 405416. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:35:41,850][00556] Avg episode reward: [(0, '21.869')] [2024-09-05 09:35:44,821][15095] Updated weights for policy 0, policy_version 1381 (0.0032) [2024-09-05 09:35:46,845][00556] Fps is (10 sec: 4096.1, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 5664768. Throughput: 0: 960.8. Samples: 412290. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:35:46,850][00556] Avg episode reward: [(0, '22.499')] [2024-09-05 09:35:51,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 5685248. Throughput: 0: 993.0. Samples: 415846. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:35:51,849][00556] Avg episode reward: [(0, '21.302')] [2024-09-05 09:35:51,863][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001388_5685248.pth... [2024-09-05 09:35:52,038][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001167_4780032.pth [2024-09-05 09:35:55,555][15095] Updated weights for policy 0, policy_version 1391 (0.0026) [2024-09-05 09:35:56,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 5697536. Throughput: 0: 947.6. Samples: 420468. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2024-09-05 09:35:56,849][00556] Avg episode reward: [(0, '20.928')] [2024-09-05 09:36:01,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3823.1, 300 sec: 3762.8). Total num frames: 5722112. Throughput: 0: 940.9. Samples: 426366. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:36:01,852][00556] Avg episode reward: [(0, '20.765')] [2024-09-05 09:36:05,354][15095] Updated weights for policy 0, policy_version 1401 (0.0037) [2024-09-05 09:36:06,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 5742592. Throughput: 0: 969.3. Samples: 429720. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:36:06,853][00556] Avg episode reward: [(0, '21.896')] [2024-09-05 09:36:11,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3776.6). Total num frames: 5758976. Throughput: 0: 976.3. Samples: 435544. Policy #0 lag: (min: 0.0, avg: 0.7, max: 2.0) [2024-09-05 09:36:11,849][00556] Avg episode reward: [(0, '21.928')] [2024-09-05 09:36:16,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 5775360. Throughput: 0: 921.5. Samples: 440006. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2024-09-05 09:36:16,847][00556] Avg episode reward: [(0, '23.344')] [2024-09-05 09:36:17,355][15095] Updated weights for policy 0, policy_version 1411 (0.0037) [2024-09-05 09:36:21,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 5799936. Throughput: 0: 941.9. Samples: 443562. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2024-09-05 09:36:21,852][00556] Avg episode reward: [(0, '23.724')] [2024-09-05 09:36:25,969][15095] Updated weights for policy 0, policy_version 1421 (0.0025) [2024-09-05 09:36:26,850][00556] Fps is (10 sec: 4503.3, 60 sec: 3890.9, 300 sec: 3776.6). Total num frames: 5820416. Throughput: 0: 1002.8. Samples: 450548. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:36:26,853][00556] Avg episode reward: [(0, '24.297')] [2024-09-05 09:36:31,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 5832704. Throughput: 0: 945.5. Samples: 454836. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:36:31,847][00556] Avg episode reward: [(0, '24.494')] [2024-09-05 09:36:36,845][00556] Fps is (10 sec: 2868.7, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 5849088. Throughput: 0: 905.8. Samples: 456608. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:36:36,847][00556] Avg episode reward: [(0, '23.334')] [2024-09-05 09:36:40,439][15095] Updated weights for policy 0, policy_version 1431 (0.0058) [2024-09-05 09:36:41,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 5865472. Throughput: 0: 909.6. Samples: 461402. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:36:41,848][00556] Avg episode reward: [(0, '20.866')] [2024-09-05 09:36:46,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.1, 300 sec: 3735.0). Total num frames: 5881856. Throughput: 0: 906.4. Samples: 467154. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:36:46,847][00556] Avg episode reward: [(0, '22.262')] [2024-09-05 09:36:51,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3721.1). Total num frames: 5898240. Throughput: 0: 880.6. Samples: 469346. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:36:51,848][00556] Avg episode reward: [(0, '22.632')] [2024-09-05 09:36:52,280][15095] Updated weights for policy 0, policy_version 1441 (0.0030) [2024-09-05 09:36:56,845][00556] Fps is (10 sec: 4095.9, 60 sec: 3754.6, 300 sec: 3735.0). Total num frames: 5922816. Throughput: 0: 886.4. Samples: 475432. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:36:56,848][00556] Avg episode reward: [(0, '21.695')] [2024-09-05 09:37:00,946][15095] Updated weights for policy 0, policy_version 1451 (0.0046) [2024-09-05 09:37:01,845][00556] Fps is (10 sec: 4505.5, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 5943296. Throughput: 0: 940.8. Samples: 482340. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:37:01,848][00556] Avg episode reward: [(0, '22.489')] [2024-09-05 09:37:06,845][00556] Fps is (10 sec: 3686.5, 60 sec: 3618.1, 300 sec: 3748.9). Total num frames: 5959680. Throughput: 0: 907.9. Samples: 484418. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:37:06,851][00556] Avg episode reward: [(0, '23.388')] [2024-09-05 09:37:11,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.1, 300 sec: 3721.1). Total num frames: 5976064. Throughput: 0: 865.4. Samples: 489486. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:37:11,852][00556] Avg episode reward: [(0, '25.665')] [2024-09-05 09:37:12,697][15095] Updated weights for policy 0, policy_version 1461 (0.0034) [2024-09-05 09:37:16,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 6000640. Throughput: 0: 922.0. Samples: 496326. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:37:16,847][00556] Avg episode reward: [(0, '25.954')] [2024-09-05 09:37:21,845][00556] Fps is (10 sec: 4096.1, 60 sec: 3618.1, 300 sec: 3748.9). Total num frames: 6017024. Throughput: 0: 951.9. Samples: 499442. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:37:21,848][00556] Avg episode reward: [(0, '25.914')] [2024-09-05 09:37:23,657][15095] Updated weights for policy 0, policy_version 1471 (0.0018) [2024-09-05 09:37:26,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3550.2, 300 sec: 3748.9). Total num frames: 6033408. Throughput: 0: 938.3. Samples: 503624. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:37:26,847][00556] Avg episode reward: [(0, '25.897')] [2024-09-05 09:37:31,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3790.5). Total num frames: 6057984. Throughput: 0: 960.3. Samples: 510366. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:37:31,851][00556] Avg episode reward: [(0, '24.750')] [2024-09-05 09:37:33,484][15095] Updated weights for policy 0, policy_version 1481 (0.0038) [2024-09-05 09:37:36,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3822.9, 300 sec: 3804.4). Total num frames: 6078464. Throughput: 0: 988.7. Samples: 513838. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:37:36,848][00556] Avg episode reward: [(0, '21.766')] [2024-09-05 09:37:41,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 6090752. Throughput: 0: 962.5. Samples: 518744. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:37:41,848][00556] Avg episode reward: [(0, '21.102')] [2024-09-05 09:37:45,419][15095] Updated weights for policy 0, policy_version 1491 (0.0055) [2024-09-05 09:37:46,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 6111232. Throughput: 0: 927.4. Samples: 524074. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:37:46,851][00556] Avg episode reward: [(0, '19.691')] [2024-09-05 09:37:51,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 3790.5). Total num frames: 6135808. Throughput: 0: 957.6. Samples: 527512. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) [2024-09-05 09:37:51,852][00556] Avg episode reward: [(0, '20.754')] [2024-09-05 09:37:51,863][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001498_6135808.pth... [2024-09-05 09:37:52,015][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001276_5226496.pth [2024-09-05 09:37:54,970][15095] Updated weights for policy 0, policy_version 1501 (0.0029) [2024-09-05 09:37:56,851][00556] Fps is (10 sec: 4093.5, 60 sec: 3822.6, 300 sec: 3804.3). Total num frames: 6152192. Throughput: 0: 976.4. Samples: 533432. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:37:56,854][00556] Avg episode reward: [(0, '21.111')] [2024-09-05 09:38:01,847][00556] Fps is (10 sec: 2866.6, 60 sec: 3686.3, 300 sec: 3762.7). Total num frames: 6164480. Throughput: 0: 921.0. Samples: 537774. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:38:01,852][00556] Avg episode reward: [(0, '20.861')] [2024-09-05 09:38:06,535][15095] Updated weights for policy 0, policy_version 1511 (0.0028) [2024-09-05 09:38:06,845][00556] Fps is (10 sec: 3688.6, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 6189056. Throughput: 0: 926.8. Samples: 541150. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:38:06,850][00556] Avg episode reward: [(0, '22.873')] [2024-09-05 09:38:11,845][00556] Fps is (10 sec: 4506.6, 60 sec: 3891.2, 300 sec: 3804.4). Total num frames: 6209536. Throughput: 0: 985.7. Samples: 547980. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:38:11,848][00556] Avg episode reward: [(0, '23.056')] [2024-09-05 09:38:16,849][00556] Fps is (10 sec: 3275.5, 60 sec: 3686.2, 300 sec: 3776.6). Total num frames: 6221824. Throughput: 0: 932.6. Samples: 552336. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:38:16,858][00556] Avg episode reward: [(0, '21.617')] [2024-09-05 09:38:18,392][15095] Updated weights for policy 0, policy_version 1521 (0.0031) [2024-09-05 09:38:21,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 6242304. Throughput: 0: 911.7. Samples: 554864. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:38:21,847][00556] Avg episode reward: [(0, '21.086')] [2024-09-05 09:38:26,845][00556] Fps is (10 sec: 4507.3, 60 sec: 3891.2, 300 sec: 3790.5). Total num frames: 6266880. Throughput: 0: 955.7. Samples: 561750. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:38:26,847][00556] Avg episode reward: [(0, '23.852')] [2024-09-05 09:38:27,349][15095] Updated weights for policy 0, policy_version 1531 (0.0017) [2024-09-05 09:38:31,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3790.5). Total num frames: 6283264. Throughput: 0: 960.2. Samples: 567284. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:38:31,851][00556] Avg episode reward: [(0, '24.038')] [2024-09-05 09:38:36,845][00556] Fps is (10 sec: 3276.9, 60 sec: 3686.4, 300 sec: 3776.7). Total num frames: 6299648. Throughput: 0: 930.3. Samples: 569376. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:38:36,847][00556] Avg episode reward: [(0, '23.419')] [2024-09-05 09:38:39,273][15095] Updated weights for policy 0, policy_version 1541 (0.0031) [2024-09-05 09:38:41,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3762.8). Total num frames: 6320128. Throughput: 0: 938.3. Samples: 575648. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:38:41,847][00556] Avg episode reward: [(0, '23.686')] [2024-09-05 09:38:46,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3790.5). Total num frames: 6340608. Throughput: 0: 982.0. Samples: 581960. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:38:46,848][00556] Avg episode reward: [(0, '24.265')] [2024-09-05 09:38:49,899][15095] Updated weights for policy 0, policy_version 1551 (0.0041) [2024-09-05 09:38:51,850][00556] Fps is (10 sec: 3684.5, 60 sec: 3686.1, 300 sec: 3776.6). Total num frames: 6356992. Throughput: 0: 952.7. Samples: 584026. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:38:51,855][00556] Avg episode reward: [(0, '22.802')] [2024-09-05 09:38:56,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3755.0, 300 sec: 3776.7). Total num frames: 6377472. Throughput: 0: 913.3. Samples: 589078. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:38:56,847][00556] Avg episode reward: [(0, '22.264')] [2024-09-05 09:39:00,436][15095] Updated weights for policy 0, policy_version 1561 (0.0021) [2024-09-05 09:39:01,845][00556] Fps is (10 sec: 4098.1, 60 sec: 3891.3, 300 sec: 3776.7). Total num frames: 6397952. Throughput: 0: 966.1. Samples: 595806. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:39:01,848][00556] Avg episode reward: [(0, '22.712')] [2024-09-05 09:39:06,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3790.5). Total num frames: 6414336. Throughput: 0: 974.7. Samples: 598724. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:39:06,849][00556] Avg episode reward: [(0, '23.303')] [2024-09-05 09:39:11,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3762.8). Total num frames: 6430720. Throughput: 0: 915.8. Samples: 602960. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:39:11,852][00556] Avg episode reward: [(0, '22.699')] [2024-09-05 09:39:12,164][15095] Updated weights for policy 0, policy_version 1571 (0.0042) [2024-09-05 09:39:16,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.5, 300 sec: 3762.8). Total num frames: 6455296. Throughput: 0: 937.8. Samples: 609484. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:39:16,848][00556] Avg episode reward: [(0, '23.351')] [2024-09-05 09:39:21,731][15095] Updated weights for policy 0, policy_version 1581 (0.0028) [2024-09-05 09:39:21,845][00556] Fps is (10 sec: 4505.5, 60 sec: 3891.2, 300 sec: 3790.6). Total num frames: 6475776. Throughput: 0: 967.5. Samples: 612914. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:39:21,849][00556] Avg episode reward: [(0, '23.398')] [2024-09-05 09:39:26,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3776.7). Total num frames: 6488064. Throughput: 0: 933.2. Samples: 617640. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:39:26,847][00556] Avg episode reward: [(0, '24.256')] [2024-09-05 09:39:31,845][00556] Fps is (10 sec: 3276.9, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 6508544. Throughput: 0: 919.3. Samples: 623330. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:39:31,848][00556] Avg episode reward: [(0, '23.903')] [2024-09-05 09:39:33,158][15095] Updated weights for policy 0, policy_version 1591 (0.0024) [2024-09-05 09:39:36,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 6533120. Throughput: 0: 950.8. Samples: 626808. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:39:36,848][00556] Avg episode reward: [(0, '25.854')] [2024-09-05 09:39:41,852][00556] Fps is (10 sec: 4093.3, 60 sec: 3822.5, 300 sec: 3790.4). Total num frames: 6549504. Throughput: 0: 969.0. Samples: 632690. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:39:41,856][00556] Avg episode reward: [(0, '26.220')] [2024-09-05 09:39:44,308][15095] Updated weights for policy 0, policy_version 1601 (0.0021) [2024-09-05 09:39:46,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 6565888. Throughput: 0: 921.1. Samples: 637256. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:39:46,848][00556] Avg episode reward: [(0, '26.206')] [2024-09-05 09:39:51,845][00556] Fps is (10 sec: 3688.8, 60 sec: 3823.3, 300 sec: 3762.8). Total num frames: 6586368. Throughput: 0: 931.4. Samples: 640636. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:39:51,850][00556] Avg episode reward: [(0, '25.493')] [2024-09-05 09:39:51,861][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001608_6586368.pth... [2024-09-05 09:39:51,995][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001388_5685248.pth [2024-09-05 09:39:54,186][15095] Updated weights for policy 0, policy_version 1611 (0.0045) [2024-09-05 09:39:56,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 6606848. Throughput: 0: 983.5. Samples: 647216. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:39:56,852][00556] Avg episode reward: [(0, '25.076')] [2024-09-05 09:40:01,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3762.8). Total num frames: 6619136. Throughput: 0: 932.9. Samples: 651464. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:40:01,851][00556] Avg episode reward: [(0, '25.335')] [2024-09-05 09:40:06,209][15095] Updated weights for policy 0, policy_version 1621 (0.0020) [2024-09-05 09:40:06,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 6639616. Throughput: 0: 914.8. Samples: 654078. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:40:06,848][00556] Avg episode reward: [(0, '25.535')] [2024-09-05 09:40:11,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 6664192. Throughput: 0: 960.4. Samples: 660858. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:40:11,847][00556] Avg episode reward: [(0, '25.321')] [2024-09-05 09:40:16,316][15095] Updated weights for policy 0, policy_version 1631 (0.0030) [2024-09-05 09:40:16,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3754.7, 300 sec: 3776.7). Total num frames: 6680576. Throughput: 0: 950.6. Samples: 666108. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:40:16,855][00556] Avg episode reward: [(0, '24.456')] [2024-09-05 09:40:21,846][00556] Fps is (10 sec: 3276.7, 60 sec: 3686.4, 300 sec: 3762.8). Total num frames: 6696960. Throughput: 0: 918.9. Samples: 668160. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) [2024-09-05 09:40:21,848][00556] Avg episode reward: [(0, '23.523')] [2024-09-05 09:40:26,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3762.8). Total num frames: 6717440. Throughput: 0: 930.1. Samples: 674540. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:40:26,848][00556] Avg episode reward: [(0, '23.357')] [2024-09-05 09:40:27,262][15095] Updated weights for policy 0, policy_version 1641 (0.0029) [2024-09-05 09:40:31,848][00556] Fps is (10 sec: 4095.9, 60 sec: 3822.9, 300 sec: 3776.6). Total num frames: 6737920. Throughput: 0: 970.7. Samples: 680936. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:40:31,851][00556] Avg episode reward: [(0, '24.016')] [2024-09-05 09:40:36,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.1, 300 sec: 3762.8). Total num frames: 6750208. Throughput: 0: 933.3. Samples: 682634. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:40:36,848][00556] Avg episode reward: [(0, '24.016')] [2024-09-05 09:40:41,325][15095] Updated weights for policy 0, policy_version 1651 (0.0026) [2024-09-05 09:40:41,845][00556] Fps is (10 sec: 2457.7, 60 sec: 3550.2, 300 sec: 3721.1). Total num frames: 6762496. Throughput: 0: 862.1. Samples: 686010. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) [2024-09-05 09:40:41,848][00556] Avg episode reward: [(0, '22.686')] [2024-09-05 09:40:46,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3618.1, 300 sec: 3721.1). Total num frames: 6782976. Throughput: 0: 899.4. Samples: 691938. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:40:46,847][00556] Avg episode reward: [(0, '24.385')] [2024-09-05 09:40:50,708][15095] Updated weights for policy 0, policy_version 1661 (0.0031) [2024-09-05 09:40:51,849][00556] Fps is (10 sec: 4503.9, 60 sec: 3686.2, 300 sec: 3762.7). Total num frames: 6807552. Throughput: 0: 917.8. Samples: 695382. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:40:51,856][00556] Avg episode reward: [(0, '27.244')] [2024-09-05 09:40:51,867][15082] Saving new best policy, reward=27.244! [2024-09-05 09:40:56,846][00556] Fps is (10 sec: 3686.0, 60 sec: 3549.8, 300 sec: 3721.1). Total num frames: 6819840. Throughput: 0: 877.9. Samples: 700366. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) [2024-09-05 09:40:56,848][00556] Avg episode reward: [(0, '26.505')] [2024-09-05 09:41:01,845][00556] Fps is (10 sec: 2868.3, 60 sec: 3618.1, 300 sec: 3707.2). Total num frames: 6836224. Throughput: 0: 879.3. Samples: 705676. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:41:01,851][00556] Avg episode reward: [(0, '25.589')] [2024-09-05 09:41:02,901][15095] Updated weights for policy 0, policy_version 1671 (0.0021) [2024-09-05 09:41:06,845][00556] Fps is (10 sec: 4096.4, 60 sec: 3686.4, 300 sec: 3735.0). Total num frames: 6860800. Throughput: 0: 910.0. Samples: 709108. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:06,851][00556] Avg episode reward: [(0, '25.582')] [2024-09-05 09:41:11,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3735.0). Total num frames: 6877184. Throughput: 0: 903.1. Samples: 715180. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:41:11,850][00556] Avg episode reward: [(0, '25.430')] [2024-09-05 09:41:13,658][15095] Updated weights for policy 0, policy_version 1681 (0.0020) [2024-09-05 09:41:16,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3707.2). Total num frames: 6893568. Throughput: 0: 853.4. Samples: 719338. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:41:16,852][00556] Avg episode reward: [(0, '24.194')] [2024-09-05 09:41:21,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3618.2, 300 sec: 3707.3). Total num frames: 6914048. Throughput: 0: 888.3. Samples: 722606. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:21,847][00556] Avg episode reward: [(0, '23.539')] [2024-09-05 09:41:23,899][15095] Updated weights for policy 0, policy_version 1691 (0.0037) [2024-09-05 09:41:26,850][00556] Fps is (10 sec: 4503.5, 60 sec: 3686.1, 300 sec: 3748.8). Total num frames: 6938624. Throughput: 0: 967.1. Samples: 729532. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:26,852][00556] Avg episode reward: [(0, '24.841')] [2024-09-05 09:41:31,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3549.9, 300 sec: 3735.0). Total num frames: 6950912. Throughput: 0: 935.6. Samples: 734040. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:31,850][00556] Avg episode reward: [(0, '23.760')] [2024-09-05 09:41:35,652][15095] Updated weights for policy 0, policy_version 1701 (0.0031) [2024-09-05 09:41:36,845][00556] Fps is (10 sec: 3278.3, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 6971392. Throughput: 0: 914.6. Samples: 736536. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:36,850][00556] Avg episode reward: [(0, '24.123')] [2024-09-05 09:41:41,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3823.0, 300 sec: 3762.8). Total num frames: 6991872. Throughput: 0: 956.1. Samples: 743390. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:41,851][00556] Avg episode reward: [(0, '25.368')] [2024-09-05 09:41:45,111][15095] Updated weights for policy 0, policy_version 1711 (0.0034) [2024-09-05 09:41:46,846][00556] Fps is (10 sec: 4095.9, 60 sec: 3822.9, 300 sec: 3776.6). Total num frames: 7012352. Throughput: 0: 964.7. Samples: 749086. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:41:46,848][00556] Avg episode reward: [(0, '24.280')] [2024-09-05 09:41:51,845][00556] Fps is (10 sec: 3276.7, 60 sec: 3618.4, 300 sec: 3735.0). Total num frames: 7024640. Throughput: 0: 933.5. Samples: 751116. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:41:51,852][00556] Avg episode reward: [(0, '24.529')] [2024-09-05 09:41:51,863][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001715_7024640.pth... [2024-09-05 09:41:51,995][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001498_6135808.pth [2024-09-05 09:41:56,518][15095] Updated weights for policy 0, policy_version 1721 (0.0036) [2024-09-05 09:41:56,845][00556] Fps is (10 sec: 3686.6, 60 sec: 3823.0, 300 sec: 3748.9). Total num frames: 7049216. Throughput: 0: 932.9. Samples: 757162. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:41:56,853][00556] Avg episode reward: [(0, '24.539')] [2024-09-05 09:42:01,845][00556] Fps is (10 sec: 4505.7, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 7069696. Throughput: 0: 993.6. Samples: 764050. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:42:01,851][00556] Avg episode reward: [(0, '24.779')] [2024-09-05 09:42:06,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3762.8). Total num frames: 7086080. Throughput: 0: 967.6. Samples: 766146. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:42:06,850][00556] Avg episode reward: [(0, '25.192')] [2024-09-05 09:42:08,005][15095] Updated weights for policy 0, policy_version 1731 (0.0037) [2024-09-05 09:42:11,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 7102464. Throughput: 0: 921.9. Samples: 771012. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:42:11,852][00556] Avg episode reward: [(0, '25.235')] [2024-09-05 09:42:16,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 7127040. Throughput: 0: 975.6. Samples: 777944. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:42:16,851][00556] Avg episode reward: [(0, '25.494')] [2024-09-05 09:42:17,272][15095] Updated weights for policy 0, policy_version 1741 (0.0022) [2024-09-05 09:42:21,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3762.8). Total num frames: 7143424. Throughput: 0: 987.6. Samples: 780976. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:42:21,850][00556] Avg episode reward: [(0, '24.848')] [2024-09-05 09:42:26,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.7, 300 sec: 3735.0). Total num frames: 7159808. Throughput: 0: 926.8. Samples: 785096. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:42:26,851][00556] Avg episode reward: [(0, '25.336')] [2024-09-05 09:42:29,189][15095] Updated weights for policy 0, policy_version 1751 (0.0029) [2024-09-05 09:42:31,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3735.0). Total num frames: 7180288. Throughput: 0: 947.8. Samples: 791736. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:42:31,850][00556] Avg episode reward: [(0, '25.558')] [2024-09-05 09:42:36,849][00556] Fps is (10 sec: 4503.7, 60 sec: 3891.0, 300 sec: 3776.6). Total num frames: 7204864. Throughput: 0: 978.0. Samples: 795130. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:42:36,852][00556] Avg episode reward: [(0, '25.976')] [2024-09-05 09:42:39,424][15095] Updated weights for policy 0, policy_version 1761 (0.0031) [2024-09-05 09:42:41,848][00556] Fps is (10 sec: 3685.5, 60 sec: 3754.5, 300 sec: 3748.8). Total num frames: 7217152. Throughput: 0: 952.5. Samples: 800028. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:42:41,855][00556] Avg episode reward: [(0, '26.972')] [2024-09-05 09:42:46,845][00556] Fps is (10 sec: 3278.2, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 7237632. Throughput: 0: 920.0. Samples: 805450. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:42:46,851][00556] Avg episode reward: [(0, '26.437')] [2024-09-05 09:42:50,266][15095] Updated weights for policy 0, policy_version 1771 (0.0029) [2024-09-05 09:42:51,845][00556] Fps is (10 sec: 4097.0, 60 sec: 3891.2, 300 sec: 3749.0). Total num frames: 7258112. Throughput: 0: 945.3. Samples: 808686. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:42:51,847][00556] Avg episode reward: [(0, '27.632')] [2024-09-05 09:42:51,861][15082] Saving new best policy, reward=27.632! [2024-09-05 09:42:56,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 7278592. Throughput: 0: 972.4. Samples: 814770. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:42:56,850][00556] Avg episode reward: [(0, '27.761')] [2024-09-05 09:42:56,854][15082] Saving new best policy, reward=27.761! [2024-09-05 09:43:01,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3735.0). Total num frames: 7290880. Throughput: 0: 914.5. Samples: 819096. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:43:01,848][00556] Avg episode reward: [(0, '27.259')] [2024-09-05 09:43:02,095][15095] Updated weights for policy 0, policy_version 1781 (0.0034) [2024-09-05 09:43:06,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3822.9, 300 sec: 3748.9). Total num frames: 7315456. Throughput: 0: 923.2. Samples: 822520. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:43:06,849][00556] Avg episode reward: [(0, '26.963')] [2024-09-05 09:43:11,207][15095] Updated weights for policy 0, policy_version 1791 (0.0034) [2024-09-05 09:43:11,846][00556] Fps is (10 sec: 4505.4, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 7335936. Throughput: 0: 982.4. Samples: 829304. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:43:11,852][00556] Avg episode reward: [(0, '26.581')] [2024-09-05 09:43:16,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 7348224. Throughput: 0: 932.3. Samples: 833690. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:43:16,848][00556] Avg episode reward: [(0, '27.727')] [2024-09-05 09:43:21,846][00556] Fps is (10 sec: 3276.7, 60 sec: 3754.6, 300 sec: 3735.0). Total num frames: 7368704. Throughput: 0: 909.2. Samples: 836042. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:43:21,849][00556] Avg episode reward: [(0, '27.547')] [2024-09-05 09:43:23,323][15095] Updated weights for policy 0, policy_version 1801 (0.0040) [2024-09-05 09:43:26,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 7393280. Throughput: 0: 953.7. Samples: 842940. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:43:26,847][00556] Avg episode reward: [(0, '26.212')] [2024-09-05 09:43:31,848][00556] Fps is (10 sec: 4095.3, 60 sec: 3822.8, 300 sec: 3762.7). Total num frames: 7409664. Throughput: 0: 958.0. Samples: 848562. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:43:31,852][00556] Avg episode reward: [(0, '25.599')] [2024-09-05 09:43:34,269][15095] Updated weights for policy 0, policy_version 1811 (0.0033) [2024-09-05 09:43:36,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.7, 300 sec: 3748.9). Total num frames: 7426048. Throughput: 0: 933.4. Samples: 850688. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:43:36,851][00556] Avg episode reward: [(0, '24.550')] [2024-09-05 09:43:41,845][00556] Fps is (10 sec: 3687.3, 60 sec: 3823.1, 300 sec: 3748.9). Total num frames: 7446528. Throughput: 0: 935.6. Samples: 856874. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:43:41,848][00556] Avg episode reward: [(0, '24.875')] [2024-09-05 09:43:43,923][15095] Updated weights for policy 0, policy_version 1821 (0.0032) [2024-09-05 09:43:46,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3891.2, 300 sec: 3776.7). Total num frames: 7471104. Throughput: 0: 991.2. Samples: 863698. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:43:46,848][00556] Avg episode reward: [(0, '25.754')] [2024-09-05 09:43:51,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 7483392. Throughput: 0: 960.3. Samples: 865734. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:43:51,847][00556] Avg episode reward: [(0, '25.042')] [2024-09-05 09:43:51,860][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001827_7483392.pth... [2024-09-05 09:43:52,041][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001608_6586368.pth [2024-09-05 09:43:55,680][15095] Updated weights for policy 0, policy_version 1831 (0.0043) [2024-09-05 09:43:56,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 7503872. Throughput: 0: 922.4. Samples: 870810. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:43:56,850][00556] Avg episode reward: [(0, '26.128')] [2024-09-05 09:44:01,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 7524352. Throughput: 0: 977.9. Samples: 877694. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:44:01,850][00556] Avg episode reward: [(0, '26.850')] [2024-09-05 09:44:05,255][15095] Updated weights for policy 0, policy_version 1841 (0.0024) [2024-09-05 09:44:06,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3822.9, 300 sec: 3776.7). Total num frames: 7544832. Throughput: 0: 991.7. Samples: 880668. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:44:06,847][00556] Avg episode reward: [(0, '26.168')] [2024-09-05 09:44:11,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3735.0). Total num frames: 7557120. Throughput: 0: 931.4. Samples: 884854. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:44:11,848][00556] Avg episode reward: [(0, '25.727')] [2024-09-05 09:44:16,476][15095] Updated weights for policy 0, policy_version 1851 (0.0038) [2024-09-05 09:44:16,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3748.9). Total num frames: 7581696. Throughput: 0: 952.9. Samples: 891442. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:44:16,852][00556] Avg episode reward: [(0, '23.619')] [2024-09-05 09:44:21,845][00556] Fps is (10 sec: 4505.5, 60 sec: 3891.2, 300 sec: 3776.6). Total num frames: 7602176. Throughput: 0: 978.9. Samples: 894738. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:44:21,848][00556] Avg episode reward: [(0, '24.323')] [2024-09-05 09:44:26,846][00556] Fps is (10 sec: 3276.4, 60 sec: 3686.3, 300 sec: 3748.9). Total num frames: 7614464. Throughput: 0: 947.9. Samples: 899532. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:44:26,857][00556] Avg episode reward: [(0, '23.888')] [2024-09-05 09:44:28,448][15095] Updated weights for policy 0, policy_version 1861 (0.0039) [2024-09-05 09:44:31,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3754.8, 300 sec: 3735.0). Total num frames: 7634944. Throughput: 0: 920.4. Samples: 905118. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:44:31,851][00556] Avg episode reward: [(0, '25.102')] [2024-09-05 09:44:36,845][00556] Fps is (10 sec: 3686.8, 60 sec: 3754.7, 300 sec: 3735.1). Total num frames: 7651328. Throughput: 0: 929.8. Samples: 907576. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:44:36,852][00556] Avg episode reward: [(0, '25.657')] [2024-09-05 09:44:41,298][15095] Updated weights for policy 0, policy_version 1871 (0.0034) [2024-09-05 09:44:41,847][00556] Fps is (10 sec: 2866.5, 60 sec: 3618.0, 300 sec: 3721.1). Total num frames: 7663616. Throughput: 0: 902.7. Samples: 911434. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:44:41,850][00556] Avg episode reward: [(0, '24.660')] [2024-09-05 09:44:46,845][00556] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3693.3). Total num frames: 7675904. Throughput: 0: 844.0. Samples: 915676. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:44:46,847][00556] Avg episode reward: [(0, '25.863')] [2024-09-05 09:44:51,845][00556] Fps is (10 sec: 3687.3, 60 sec: 3618.1, 300 sec: 3707.2). Total num frames: 7700480. Throughput: 0: 850.8. Samples: 918952. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:44:51,848][00556] Avg episode reward: [(0, '26.791')] [2024-09-05 09:44:52,492][15095] Updated weights for policy 0, policy_version 1881 (0.0033) [2024-09-05 09:44:56,845][00556] Fps is (10 sec: 4505.6, 60 sec: 3618.1, 300 sec: 3735.0). Total num frames: 7720960. Throughput: 0: 905.0. Samples: 925580. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) [2024-09-05 09:44:56,848][00556] Avg episode reward: [(0, '26.411')] [2024-09-05 09:45:01,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3549.9, 300 sec: 3721.1). Total num frames: 7737344. Throughput: 0: 862.4. Samples: 930252. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:01,852][00556] Avg episode reward: [(0, '26.766')] [2024-09-05 09:45:04,418][15095] Updated weights for policy 0, policy_version 1891 (0.0055) [2024-09-05 09:45:06,845][00556] Fps is (10 sec: 3276.7, 60 sec: 3481.6, 300 sec: 3693.3). Total num frames: 7753728. Throughput: 0: 840.4. Samples: 932554. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:06,851][00556] Avg episode reward: [(0, '26.762')] [2024-09-05 09:45:11,845][00556] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3721.1). Total num frames: 7778304. Throughput: 0: 887.2. Samples: 939454. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) [2024-09-05 09:45:11,848][00556] Avg episode reward: [(0, '27.294')] [2024-09-05 09:45:13,326][15095] Updated weights for policy 0, policy_version 1901 (0.0023) [2024-09-05 09:45:16,845][00556] Fps is (10 sec: 4096.1, 60 sec: 3549.9, 300 sec: 3721.1). Total num frames: 7794688. Throughput: 0: 889.6. Samples: 945150. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:16,851][00556] Avg episode reward: [(0, '27.508')] [2024-09-05 09:45:21,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3481.6, 300 sec: 3707.2). Total num frames: 7811072. Throughput: 0: 881.7. Samples: 947252. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) [2024-09-05 09:45:21,852][00556] Avg episode reward: [(0, '28.033')] [2024-09-05 09:45:21,864][15082] Saving new best policy, reward=28.033! [2024-09-05 09:45:25,366][15095] Updated weights for policy 0, policy_version 1911 (0.0024) [2024-09-05 09:45:26,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3618.2, 300 sec: 3707.2). Total num frames: 7831552. Throughput: 0: 926.0. Samples: 953102. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:26,852][00556] Avg episode reward: [(0, '26.995')] [2024-09-05 09:45:31,845][00556] Fps is (10 sec: 4095.9, 60 sec: 3618.1, 300 sec: 3735.0). Total num frames: 7852032. Throughput: 0: 983.3. Samples: 959924. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:31,851][00556] Avg episode reward: [(0, '26.897')] [2024-09-05 09:45:36,176][15095] Updated weights for policy 0, policy_version 1921 (0.0027) [2024-09-05 09:45:36,849][00556] Fps is (10 sec: 3684.9, 60 sec: 3617.9, 300 sec: 3748.8). Total num frames: 7868416. Throughput: 0: 956.1. Samples: 961980. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:36,854][00556] Avg episode reward: [(0, '26.159')] [2024-09-05 09:45:41,845][00556] Fps is (10 sec: 3686.5, 60 sec: 3754.8, 300 sec: 3748.9). Total num frames: 7888896. Throughput: 0: 916.9. Samples: 966842. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:41,852][00556] Avg episode reward: [(0, '27.208')] [2024-09-05 09:45:46,242][15095] Updated weights for policy 0, policy_version 1931 (0.0029) [2024-09-05 09:45:46,845][00556] Fps is (10 sec: 4097.7, 60 sec: 3891.2, 300 sec: 3735.0). Total num frames: 7909376. Throughput: 0: 964.3. Samples: 973646. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:46,851][00556] Avg episode reward: [(0, '25.253')] [2024-09-05 09:45:51,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3748.9). Total num frames: 7925760. Throughput: 0: 983.7. Samples: 976822. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:51,847][00556] Avg episode reward: [(0, '25.697')] [2024-09-05 09:45:51,865][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001935_7925760.pth... [2024-09-05 09:45:52,055][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001715_7024640.pth [2024-09-05 09:45:56,845][00556] Fps is (10 sec: 3276.8, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 7942144. Throughput: 0: 920.3. Samples: 980866. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:45:56,847][00556] Avg episode reward: [(0, '25.991')] [2024-09-05 09:45:58,330][15095] Updated weights for policy 0, policy_version 1941 (0.0026) [2024-09-05 09:46:01,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3754.7, 300 sec: 3735.0). Total num frames: 7962624. Throughput: 0: 937.1. Samples: 987318. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:46:01,847][00556] Avg episode reward: [(0, '26.448')] [2024-09-05 09:46:06,845][00556] Fps is (10 sec: 4505.5, 60 sec: 3891.2, 300 sec: 3762.8). Total num frames: 7987200. Throughput: 0: 967.3. Samples: 990780. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) [2024-09-05 09:46:06,847][00556] Avg episode reward: [(0, '27.241')] [2024-09-05 09:46:07,783][15095] Updated weights for policy 0, policy_version 1951 (0.0022) [2024-09-05 09:46:11,845][00556] Fps is (10 sec: 3686.4, 60 sec: 3686.4, 300 sec: 3748.9). Total num frames: 7999488. Throughput: 0: 949.2. Samples: 995818. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) [2024-09-05 09:46:11,852][00556] Avg episode reward: [(0, '27.430')] [2024-09-05 09:46:13,592][15082] Stopping Batcher_0... [2024-09-05 09:46:13,593][15082] Loop batcher_evt_loop terminating... [2024-09-05 09:46:13,593][00556] Component Batcher_0 stopped! [2024-09-05 09:46:13,602][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001955_8007680.pth... [2024-09-05 09:46:13,692][15095] Weights refcount: 2 0 [2024-09-05 09:46:13,698][15095] Stopping InferenceWorker_p0-w0... [2024-09-05 09:46:13,698][00556] Component InferenceWorker_p0-w0 stopped! [2024-09-05 09:46:13,707][15095] Loop inference_proc0-0_evt_loop terminating... [2024-09-05 09:46:13,758][15082] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001827_7483392.pth [2024-09-05 09:46:13,777][15082] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001955_8007680.pth... [2024-09-05 09:46:13,998][00556] Component LearnerWorker_p0 stopped! [2024-09-05 09:46:14,002][15082] Stopping LearnerWorker_p0... [2024-09-05 09:46:14,003][15082] Loop learner_proc0_evt_loop terminating... [2024-09-05 09:46:14,097][15102] Stopping RolloutWorker_w6... [2024-09-05 09:46:14,098][15102] Loop rollout_proc6_evt_loop terminating... [2024-09-05 09:46:14,097][00556] Component RolloutWorker_w6 stopped! [2024-09-05 09:46:14,115][00556] Component RolloutWorker_w4 stopped! [2024-09-05 09:46:14,115][15099] Stopping RolloutWorker_w4... [2024-09-05 09:46:14,120][15099] Loop rollout_proc4_evt_loop terminating... [2024-09-05 09:46:14,129][15096] Stopping RolloutWorker_w0... [2024-09-05 09:46:14,129][00556] Component RolloutWorker_w0 stopped! [2024-09-05 09:46:14,131][15096] Loop rollout_proc0_evt_loop terminating... [2024-09-05 09:46:14,185][15100] Stopping RolloutWorker_w2... [2024-09-05 09:46:14,185][00556] Component RolloutWorker_w2 stopped! [2024-09-05 09:46:14,186][15100] Loop rollout_proc2_evt_loop terminating... [2024-09-05 09:46:14,260][00556] Component RolloutWorker_w5 stopped! [2024-09-05 09:46:14,263][15101] Stopping RolloutWorker_w5... [2024-09-05 09:46:14,264][15101] Loop rollout_proc5_evt_loop terminating... [2024-09-05 09:46:14,300][00556] Component RolloutWorker_w1 stopped! [2024-09-05 09:46:14,305][15097] Stopping RolloutWorker_w1... [2024-09-05 09:46:14,316][15097] Loop rollout_proc1_evt_loop terminating... [2024-09-05 09:46:14,319][00556] Component RolloutWorker_w7 stopped! [2024-09-05 09:46:14,322][15103] Stopping RolloutWorker_w7... [2024-09-05 09:46:14,332][15103] Loop rollout_proc7_evt_loop terminating... [2024-09-05 09:46:14,343][00556] Component RolloutWorker_w3 stopped! [2024-09-05 09:46:14,346][00556] Waiting for process learner_proc0 to stop... [2024-09-05 09:46:14,349][15098] Stopping RolloutWorker_w3... [2024-09-05 09:46:14,350][15098] Loop rollout_proc3_evt_loop terminating... [2024-09-05 09:46:15,506][00556] Waiting for process inference_proc0-0 to join... [2024-09-05 09:46:15,514][00556] Waiting for process rollout_proc0 to join... [2024-09-05 09:46:17,511][00556] Waiting for process rollout_proc1 to join... [2024-09-05 09:46:17,604][00556] Waiting for process rollout_proc2 to join... [2024-09-05 09:46:17,608][00556] Waiting for process rollout_proc3 to join... [2024-09-05 09:46:17,611][00556] Waiting for process rollout_proc4 to join... [2024-09-05 09:46:17,616][00556] Waiting for process rollout_proc5 to join... [2024-09-05 09:46:17,619][00556] Waiting for process rollout_proc6 to join... [2024-09-05 09:46:17,623][00556] Waiting for process rollout_proc7 to join... [2024-09-05 09:46:17,627][00556] Batcher 0 profile tree view: batching: 27.8309, releasing_batches: 0.0254 [2024-09-05 09:46:17,630][00556] InferenceWorker_p0-w0 profile tree view: wait_policy: 0.0000 wait_policy_total: 377.7299 update_model: 9.6648 weight_update: 0.0022 one_step: 0.0238 handle_policy_step: 644.3478 deserialize: 15.4863, stack: 3.3304, obs_to_device_normalize: 130.8669, forward: 340.9284, send_messages: 31.0169 prepare_outputs: 91.3202 to_cpu: 53.1027 [2024-09-05 09:46:17,632][00556] Learner 0 profile tree view: misc: 0.0061, prepare_batch: 13.8632 train: 76.2424 epoch_init: 0.0142, minibatch_init: 0.0131, losses_postprocess: 0.7158, kl_divergence: 0.7868, after_optimizer: 3.3525 calculate_losses: 27.2501 losses_init: 0.0039, forward_head: 1.4150, bptt_initial: 18.2483, tail: 1.1865, advantages_returns: 0.2817, losses: 3.8163 bptt: 1.9909 bptt_forward_core: 1.9021 update: 43.4576 clip: 0.8827 [2024-09-05 09:46:17,634][00556] RolloutWorker_w0 profile tree view: wait_for_trajectories: 0.3461, enqueue_policy_requests: 95.0183, env_step: 836.8298, overhead: 13.5378, complete_rollouts: 7.2882 save_policy_outputs: 22.1033 split_output_tensors: 8.5073 [2024-09-05 09:46:17,637][00556] RolloutWorker_w7 profile tree view: wait_for_trajectories: 0.3294, enqueue_policy_requests: 98.0879, env_step: 835.8569, overhead: 14.6034, complete_rollouts: 6.7993 save_policy_outputs: 21.8937 split_output_tensors: 9.0379 [2024-09-05 09:46:17,638][00556] Loop Runner_EvtLoop terminating... [2024-09-05 09:46:17,639][00556] Runner profile tree view: main_loop: 1103.4731 [2024-09-05 09:46:17,640][00556] Collected {0: 8007680}, FPS: 3615.4 [2024-09-05 09:46:17,668][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:46:17,670][00556] Overriding arg 'num_workers' with value 1 passed from command line [2024-09-05 09:46:17,673][00556] Adding new argument 'no_render'=True that is not in the saved config file! [2024-09-05 09:46:17,675][00556] Adding new argument 'save_video'=True that is not in the saved config file! [2024-09-05 09:46:17,677][00556] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:46:17,678][00556] Adding new argument 'video_name'=None that is not in the saved config file! [2024-09-05 09:46:17,681][00556] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:46:17,684][00556] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2024-09-05 09:46:17,685][00556] Adding new argument 'push_to_hub'=False that is not in the saved config file! [2024-09-05 09:46:17,686][00556] Adding new argument 'hf_repository'=None that is not in the saved config file! [2024-09-05 09:46:17,687][00556] Adding new argument 'policy_index'=0 that is not in the saved config file! [2024-09-05 09:46:17,690][00556] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2024-09-05 09:46:17,691][00556] Adding new argument 'train_script'=None that is not in the saved config file! [2024-09-05 09:46:17,693][00556] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2024-09-05 09:46:17,694][00556] Using frameskip 1 and render_action_repeat=4 for evaluation [2024-09-05 09:46:17,733][00556] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:46:17,736][00556] RunningMeanStd input shape: (1,) [2024-09-05 09:46:17,752][00556] ConvEncoder: input_channels=3 [2024-09-05 09:46:17,799][00556] Conv encoder output size: 512 [2024-09-05 09:46:17,801][00556] Policy head output size: 512 [2024-09-05 09:46:17,824][00556] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001955_8007680.pth... [2024-09-05 09:46:18,262][00556] Num frames 100... [2024-09-05 09:46:18,386][00556] Num frames 200... [2024-09-05 09:46:18,519][00556] Num frames 300... [2024-09-05 09:46:18,642][00556] Num frames 400... [2024-09-05 09:46:18,770][00556] Num frames 500... [2024-09-05 09:46:18,893][00556] Num frames 600... [2024-09-05 09:46:19,017][00556] Num frames 700... [2024-09-05 09:46:19,143][00556] Num frames 800... [2024-09-05 09:46:19,272][00556] Num frames 900... [2024-09-05 09:46:19,397][00556] Num frames 1000... [2024-09-05 09:46:19,525][00556] Num frames 1100... [2024-09-05 09:46:19,651][00556] Num frames 1200... [2024-09-05 09:46:19,781][00556] Num frames 1300... [2024-09-05 09:46:19,907][00556] Num frames 1400... [2024-09-05 09:46:20,033][00556] Num frames 1500... [2024-09-05 09:46:20,122][00556] Avg episode rewards: #0: 36.280, true rewards: #0: 15.280 [2024-09-05 09:46:20,124][00556] Avg episode reward: 36.280, avg true_objective: 15.280 [2024-09-05 09:46:20,221][00556] Num frames 1600... [2024-09-05 09:46:20,358][00556] Num frames 1700... [2024-09-05 09:46:20,486][00556] Num frames 1800... [2024-09-05 09:46:20,611][00556] Num frames 1900... [2024-09-05 09:46:20,781][00556] Num frames 2000... [2024-09-05 09:46:20,911][00556] Num frames 2100... [2024-09-05 09:46:21,039][00556] Num frames 2200... [2024-09-05 09:46:21,163][00556] Num frames 2300... [2024-09-05 09:46:21,340][00556] Avg episode rewards: #0: 26.960, true rewards: #0: 11.960 [2024-09-05 09:46:21,342][00556] Avg episode reward: 26.960, avg true_objective: 11.960 [2024-09-05 09:46:21,356][00556] Num frames 2400... [2024-09-05 09:46:21,483][00556] Num frames 2500... [2024-09-05 09:46:21,609][00556] Num frames 2600... [2024-09-05 09:46:21,740][00556] Num frames 2700... [2024-09-05 09:46:21,862][00556] Num frames 2800... [2024-09-05 09:46:21,986][00556] Num frames 2900... [2024-09-05 09:46:22,114][00556] Num frames 3000... [2024-09-05 09:46:22,242][00556] Num frames 3100... [2024-09-05 09:46:22,380][00556] Num frames 3200... [2024-09-05 09:46:22,504][00556] Num frames 3300... [2024-09-05 09:46:22,627][00556] Num frames 3400... [2024-09-05 09:46:22,760][00556] Num frames 3500... [2024-09-05 09:46:22,880][00556] Num frames 3600... [2024-09-05 09:46:23,002][00556] Num frames 3700... [2024-09-05 09:46:23,127][00556] Num frames 3800... [2024-09-05 09:46:23,253][00556] Num frames 3900... [2024-09-05 09:46:23,378][00556] Avg episode rewards: #0: 30.830, true rewards: #0: 13.163 [2024-09-05 09:46:23,380][00556] Avg episode reward: 30.830, avg true_objective: 13.163 [2024-09-05 09:46:23,441][00556] Num frames 4000... [2024-09-05 09:46:23,563][00556] Num frames 4100... [2024-09-05 09:46:23,697][00556] Num frames 4200... [2024-09-05 09:46:23,822][00556] Num frames 4300... [2024-09-05 09:46:23,982][00556] Num frames 4400... [2024-09-05 09:46:24,160][00556] Num frames 4500... [2024-09-05 09:46:24,349][00556] Avg episode rewards: #0: 26.195, true rewards: #0: 11.445 [2024-09-05 09:46:24,351][00556] Avg episode reward: 26.195, avg true_objective: 11.445 [2024-09-05 09:46:24,390][00556] Num frames 4600... [2024-09-05 09:46:24,565][00556] Num frames 4700... [2024-09-05 09:46:24,738][00556] Num frames 4800... [2024-09-05 09:46:24,900][00556] Num frames 4900... [2024-09-05 09:46:25,063][00556] Num frames 5000... [2024-09-05 09:46:25,242][00556] Num frames 5100... [2024-09-05 09:46:25,419][00556] Num frames 5200... [2024-09-05 09:46:25,597][00556] Num frames 5300... [2024-09-05 09:46:25,771][00556] Num frames 5400... [2024-09-05 09:46:25,951][00556] Num frames 5500... [2024-09-05 09:46:26,024][00556] Avg episode rewards: #0: 24.614, true rewards: #0: 11.014 [2024-09-05 09:46:26,026][00556] Avg episode reward: 24.614, avg true_objective: 11.014 [2024-09-05 09:46:26,202][00556] Num frames 5600... [2024-09-05 09:46:26,355][00556] Num frames 5700... [2024-09-05 09:46:26,488][00556] Num frames 5800... [2024-09-05 09:46:26,591][00556] Avg episode rewards: #0: 21.565, true rewards: #0: 9.732 [2024-09-05 09:46:26,593][00556] Avg episode reward: 21.565, avg true_objective: 9.732 [2024-09-05 09:46:26,673][00556] Num frames 5900... [2024-09-05 09:46:26,799][00556] Num frames 6000... [2024-09-05 09:46:26,922][00556] Num frames 6100... [2024-09-05 09:46:27,048][00556] Num frames 6200... [2024-09-05 09:46:27,170][00556] Num frames 6300... [2024-09-05 09:46:27,294][00556] Num frames 6400... [2024-09-05 09:46:27,423][00556] Num frames 6500... [2024-09-05 09:46:27,579][00556] Avg episode rewards: #0: 20.250, true rewards: #0: 9.393 [2024-09-05 09:46:27,580][00556] Avg episode reward: 20.250, avg true_objective: 9.393 [2024-09-05 09:46:27,613][00556] Num frames 6600... [2024-09-05 09:46:27,741][00556] Num frames 6700... [2024-09-05 09:46:27,863][00556] Num frames 6800... [2024-09-05 09:46:27,986][00556] Num frames 6900... [2024-09-05 09:46:28,109][00556] Num frames 7000... [2024-09-05 09:46:28,233][00556] Num frames 7100... [2024-09-05 09:46:28,359][00556] Num frames 7200... [2024-09-05 09:46:28,489][00556] Num frames 7300... [2024-09-05 09:46:28,617][00556] Num frames 7400... [2024-09-05 09:46:28,748][00556] Num frames 7500... [2024-09-05 09:46:28,871][00556] Num frames 7600... [2024-09-05 09:46:28,992][00556] Num frames 7700... [2024-09-05 09:46:29,113][00556] Num frames 7800... [2024-09-05 09:46:29,241][00556] Num frames 7900... [2024-09-05 09:46:29,320][00556] Avg episode rewards: #0: 21.399, true rewards: #0: 9.899 [2024-09-05 09:46:29,322][00556] Avg episode reward: 21.399, avg true_objective: 9.899 [2024-09-05 09:46:29,422][00556] Num frames 8000... [2024-09-05 09:46:29,552][00556] Num frames 8100... [2024-09-05 09:46:29,681][00556] Num frames 8200... [2024-09-05 09:46:29,806][00556] Num frames 8300... [2024-09-05 09:46:29,929][00556] Num frames 8400... [2024-09-05 09:46:30,048][00556] Num frames 8500... [2024-09-05 09:46:30,167][00556] Num frames 8600... [2024-09-05 09:46:30,290][00556] Num frames 8700... [2024-09-05 09:46:30,414][00556] Num frames 8800... [2024-09-05 09:46:30,548][00556] Num frames 8900... [2024-09-05 09:46:30,670][00556] Num frames 9000... [2024-09-05 09:46:30,796][00556] Num frames 9100... [2024-09-05 09:46:30,918][00556] Num frames 9200... [2024-09-05 09:46:31,042][00556] Num frames 9300... [2024-09-05 09:46:31,163][00556] Num frames 9400... [2024-09-05 09:46:31,325][00556] Avg episode rewards: #0: 23.433, true rewards: #0: 10.544 [2024-09-05 09:46:31,326][00556] Avg episode reward: 23.433, avg true_objective: 10.544 [2024-09-05 09:46:31,341][00556] Num frames 9500... [2024-09-05 09:46:31,460][00556] Num frames 9600... [2024-09-05 09:46:31,590][00556] Num frames 9700... [2024-09-05 09:46:31,721][00556] Num frames 9800... [2024-09-05 09:46:31,845][00556] Num frames 9900... [2024-09-05 09:46:31,967][00556] Num frames 10000... [2024-09-05 09:46:32,094][00556] Num frames 10100... [2024-09-05 09:46:32,217][00556] Num frames 10200... [2024-09-05 09:46:32,346][00556] Num frames 10300... [2024-09-05 09:46:32,471][00556] Avg episode rewards: #0: 23.154, true rewards: #0: 10.354 [2024-09-05 09:46:32,472][00556] Avg episode reward: 23.154, avg true_objective: 10.354 [2024-09-05 09:47:37,581][00556] Replay video saved to /content/train_dir/default_experiment/replay.mp4! [2024-09-05 09:47:38,072][00556] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json [2024-09-05 09:47:38,074][00556] Overriding arg 'num_workers' with value 1 passed from command line [2024-09-05 09:47:38,076][00556] Adding new argument 'no_render'=True that is not in the saved config file! [2024-09-05 09:47:38,079][00556] Adding new argument 'save_video'=True that is not in the saved config file! [2024-09-05 09:47:38,080][00556] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! [2024-09-05 09:47:38,082][00556] Adding new argument 'video_name'=None that is not in the saved config file! [2024-09-05 09:47:38,084][00556] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! [2024-09-05 09:47:38,085][00556] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! [2024-09-05 09:47:38,086][00556] Adding new argument 'push_to_hub'=True that is not in the saved config file! [2024-09-05 09:47:38,087][00556] Adding new argument 'hf_repository'='neeldevenshah/rl_course_vizdoom_health_gathering_supreme' that is not in the saved config file! [2024-09-05 09:47:38,088][00556] Adding new argument 'policy_index'=0 that is not in the saved config file! [2024-09-05 09:47:38,090][00556] Adding new argument 'eval_deterministic'=False that is not in the saved config file! [2024-09-05 09:47:38,091][00556] Adding new argument 'train_script'=None that is not in the saved config file! [2024-09-05 09:47:38,092][00556] Adding new argument 'enjoy_script'=None that is not in the saved config file! [2024-09-05 09:47:38,093][00556] Using frameskip 1 and render_action_repeat=4 for evaluation [2024-09-05 09:47:38,133][00556] RunningMeanStd input shape: (3, 72, 128) [2024-09-05 09:47:38,136][00556] RunningMeanStd input shape: (1,) [2024-09-05 09:47:38,153][00556] ConvEncoder: input_channels=3 [2024-09-05 09:47:38,209][00556] Conv encoder output size: 512 [2024-09-05 09:47:38,211][00556] Policy head output size: 512 [2024-09-05 09:47:38,237][00556] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000001955_8007680.pth... [2024-09-05 09:47:38,853][00556] Num frames 100... [2024-09-05 09:47:39,014][00556] Num frames 200... [2024-09-05 09:47:39,186][00556] Num frames 300... [2024-09-05 09:47:39,344][00556] Num frames 400... [2024-09-05 09:47:39,500][00556] Num frames 500... [2024-09-05 09:47:39,663][00556] Num frames 600... [2024-09-05 09:47:39,831][00556] Num frames 700... [2024-09-05 09:47:39,993][00556] Num frames 800... [2024-09-05 09:47:40,145][00556] Avg episode rewards: #0: 17.600, true rewards: #0: 8.600 [2024-09-05 09:47:40,147][00556] Avg episode reward: 17.600, avg true_objective: 8.600 [2024-09-05 09:47:40,217][00556] Num frames 900... [2024-09-05 09:47:40,387][00556] Num frames 1000... [2024-09-05 09:47:40,555][00556] Num frames 1100... [2024-09-05 09:47:40,718][00556] Num frames 1200... [2024-09-05 09:47:40,881][00556] Num frames 1300... [2024-09-05 09:47:41,058][00556] Num frames 1400... [2024-09-05 09:47:41,228][00556] Num frames 1500... [2024-09-05 09:47:41,292][00556] Avg episode rewards: #0: 14.515, true rewards: #0: 7.515 [2024-09-05 09:47:41,294][00556] Avg episode reward: 14.515, avg true_objective: 7.515 [2024-09-05 09:47:41,466][00556] Num frames 1600... [2024-09-05 09:47:41,654][00556] Num frames 1700... [2024-09-05 09:47:41,842][00556] Num frames 1800... [2024-09-05 09:47:42,030][00556] Num frames 1900... [2024-09-05 09:47:42,229][00556] Num frames 2000... [2024-09-05 09:47:42,427][00556] Num frames 2100... [2024-09-05 09:47:42,508][00556] Avg episode rewards: #0: 13.363, true rewards: #0: 7.030 [2024-09-05 09:47:42,510][00556] Avg episode reward: 13.363, avg true_objective: 7.030 [2024-09-05 09:47:42,679][00556] Num frames 2200... [2024-09-05 09:47:42,895][00556] Num frames 2300... [2024-09-05 09:47:43,103][00556] Num frames 2400... [2024-09-05 09:47:43,337][00556] Num frames 2500... [2024-09-05 09:47:43,523][00556] Num frames 2600... [2024-09-05 09:47:43,733][00556] Num frames 2700... [2024-09-05 09:47:43,949][00556] Num frames 2800... [2024-09-05 09:47:44,146][00556] Num frames 2900... [2024-09-05 09:47:44,361][00556] Num frames 3000... [2024-09-05 09:47:44,584][00556] Num frames 3100... [2024-09-05 09:47:44,842][00556] Avg episode rewards: #0: 16.243, true rewards: #0: 7.992 [2024-09-05 09:47:44,843][00556] Avg episode reward: 16.243, avg true_objective: 7.992 [2024-09-05 09:47:44,850][00556] Num frames 3200... [2024-09-05 09:47:45,060][00556] Num frames 3300... [2024-09-05 09:47:45,257][00556] Num frames 3400... [2024-09-05 09:47:45,448][00556] Num frames 3500... [2024-09-05 09:47:45,642][00556] Num frames 3600... [2024-09-05 09:47:45,842][00556] Num frames 3700... [2024-09-05 09:47:46,067][00556] Num frames 3800... [2024-09-05 09:47:46,292][00556] Num frames 3900... [2024-09-05 09:47:46,431][00556] Num frames 4000... [2024-09-05 09:47:46,524][00556] Avg episode rewards: #0: 16.658, true rewards: #0: 8.058 [2024-09-05 09:47:46,525][00556] Avg episode reward: 16.658, avg true_objective: 8.058 [2024-09-05 09:47:46,611][00556] Num frames 4100... [2024-09-05 09:47:46,742][00556] Num frames 4200... [2024-09-05 09:47:46,866][00556] Num frames 4300... [2024-09-05 09:47:46,990][00556] Num frames 4400... [2024-09-05 09:47:47,111][00556] Num frames 4500... [2024-09-05 09:47:47,260][00556] Avg episode rewards: #0: 15.122, true rewards: #0: 7.622 [2024-09-05 09:47:47,261][00556] Avg episode reward: 15.122, avg true_objective: 7.622 [2024-09-05 09:47:47,296][00556] Num frames 4600... [2024-09-05 09:47:47,419][00556] Num frames 4700... [2024-09-05 09:47:47,551][00556] Num frames 4800... [2024-09-05 09:47:47,678][00556] Num frames 4900... [2024-09-05 09:47:47,807][00556] Num frames 5000... [2024-09-05 09:47:47,929][00556] Num frames 5100... [2024-09-05 09:47:48,054][00556] Num frames 5200... [2024-09-05 09:47:48,188][00556] Num frames 5300... [2024-09-05 09:47:48,312][00556] Num frames 5400... [2024-09-05 09:47:48,440][00556] Num frames 5500... [2024-09-05 09:47:48,572][00556] Num frames 5600... [2024-09-05 09:47:48,705][00556] Num frames 5700... [2024-09-05 09:47:48,830][00556] Num frames 5800... [2024-09-05 09:47:48,951][00556] Num frames 5900... [2024-09-05 09:47:49,071][00556] Num frames 6000... [2024-09-05 09:47:49,192][00556] Num frames 6100... [2024-09-05 09:47:49,315][00556] Num frames 6200... [2024-09-05 09:47:49,441][00556] Num frames 6300... [2024-09-05 09:47:49,523][00556] Avg episode rewards: #0: 18.887, true rewards: #0: 9.030 [2024-09-05 09:47:49,525][00556] Avg episode reward: 18.887, avg true_objective: 9.030 [2024-09-05 09:47:49,628][00556] Num frames 6400... [2024-09-05 09:47:49,760][00556] Num frames 6500... [2024-09-05 09:47:49,883][00556] Num frames 6600... [2024-09-05 09:47:50,006][00556] Num frames 6700... [2024-09-05 09:47:50,129][00556] Num frames 6800... [2024-09-05 09:47:50,251][00556] Num frames 6900... [2024-09-05 09:47:50,377][00556] Num frames 7000... [2024-09-05 09:47:50,506][00556] Num frames 7100... [2024-09-05 09:47:50,637][00556] Num frames 7200... [2024-09-05 09:47:50,764][00556] Num frames 7300... [2024-09-05 09:47:50,890][00556] Num frames 7400... [2024-09-05 09:47:51,014][00556] Num frames 7500... [2024-09-05 09:47:51,139][00556] Num frames 7600... [2024-09-05 09:47:51,263][00556] Num frames 7700... [2024-09-05 09:47:51,387][00556] Num frames 7800... [2024-09-05 09:47:51,509][00556] Num frames 7900... [2024-09-05 09:47:51,649][00556] Avg episode rewards: #0: 22.336, true rewards: #0: 9.961 [2024-09-05 09:47:51,651][00556] Avg episode reward: 22.336, avg true_objective: 9.961 [2024-09-05 09:47:51,706][00556] Num frames 8000... [2024-09-05 09:47:51,836][00556] Num frames 8100... [2024-09-05 09:47:51,961][00556] Num frames 8200... [2024-09-05 09:47:52,087][00556] Num frames 8300... [2024-09-05 09:47:52,211][00556] Num frames 8400... [2024-09-05 09:47:52,341][00556] Num frames 8500... [2024-09-05 09:47:52,465][00556] Num frames 8600... [2024-09-05 09:47:52,587][00556] Num frames 8700... [2024-09-05 09:47:52,712][00556] Num frames 8800... [2024-09-05 09:47:52,848][00556] Num frames 8900... [2024-09-05 09:47:52,971][00556] Num frames 9000... [2024-09-05 09:47:53,090][00556] Num frames 9100... [2024-09-05 09:47:53,211][00556] Num frames 9200... [2024-09-05 09:47:53,333][00556] Num frames 9300... [2024-09-05 09:47:53,485][00556] Avg episode rewards: #0: 23.643, true rewards: #0: 10.421 [2024-09-05 09:47:53,486][00556] Avg episode reward: 23.643, avg true_objective: 10.421 [2024-09-05 09:47:53,514][00556] Num frames 9400... [2024-09-05 09:47:53,634][00556] Num frames 9500... [2024-09-05 09:47:53,766][00556] Num frames 9600... [2024-09-05 09:47:53,896][00556] Num frames 9700... [2024-09-05 09:47:54,018][00556] Num frames 9800... [2024-09-05 09:47:54,138][00556] Num frames 9900... [2024-09-05 09:47:54,266][00556] Avg episode rewards: #0: 22.360, true rewards: #0: 9.960 [2024-09-05 09:47:54,267][00556] Avg episode reward: 22.360, avg true_objective: 9.960 [2024-09-05 09:48:57,514][00556] Replay video saved to /content/train_dir/default_experiment/replay.mp4!