zjowowen commited on
Commit
a69b641
1 Parent(s): cf7ee90

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +302 -0
README.md ADDED
@@ -0,0 +1,302 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: apache-2.0
4
+ library_name: pytorch
5
+ tags:
6
+ - deep-reinforcement-learning
7
+ - reinforcement-learning
8
+ - DI-engine
9
+ - MsPacmanNoFrameskip-v4
10
+ benchmark_name: OpenAI/Gym/Atari
11
+ task_name: MsPacmanNoFrameskip-v4
12
+ pipeline_tag: reinforcement-learning
13
+ model-index:
14
+ - name: MuZero
15
+ results:
16
+ - task:
17
+ type: reinforcement-learning
18
+ name: reinforcement-learning
19
+ dataset:
20
+ name: MsPacmanNoFrameskip-v4
21
+ type: MsPacmanNoFrameskip-v4
22
+ metrics:
23
+ - type: mean_reward
24
+ value: 1230.0 +/- 360.53
25
+ name: mean_reward
26
+ ---
27
+
28
+ # Play **MsPacmanNoFrameskip-v4** with **MuZero** Policy
29
+
30
+ ## Model Description
31
+ <!-- Provide a longer summary of what this model is. -->
32
+
33
+ This implementation applies **MuZero** to the OpenAI/Gym/Atari **MsPacmanNoFrameskip-v4** environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine).
34
+
35
+ **LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).
36
+
37
+ ## Model Usage
38
+ ### Install the Dependencies
39
+ <details close>
40
+ <summary>(Click for Details)</summary>
41
+
42
+ ```shell
43
+ # install huggingface_ding
44
+ git clone https://github.com/opendilab/huggingface_ding.git
45
+ pip3 install -e ./huggingface_ding/
46
+ # install environment dependencies if needed
47
+
48
+ pip3 install DI-engine[common_env,video]
49
+ pip3 install LightZero
50
+
51
+ ```
52
+ </details>
53
+
54
+ ### Git Clone from Huggingface and Run the Model
55
+
56
+ <details close>
57
+ <summary>(Click for Details)</summary>
58
+
59
+ ```shell
60
+ # running with trained model
61
+ python3 -u run.py
62
+ ```
63
+ **run.py**
64
+ ```python
65
+ from lzero.agent import MuZeroAgent
66
+ from ding.config import Config
67
+ from easydict import EasyDict
68
+ import torch
69
+
70
+ # Pull model from files which are git cloned from huggingface
71
+ policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu"))
72
+ cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict)
73
+ # Instantiate the agent
74
+ agent = MuZeroAgent(
75
+ env_id="PongNoFrameskip-v4", exp_name="PongNoFrameskip-v4-MuZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
76
+ )
77
+ # Continue training
78
+ agent.train(step=5000)
79
+ # Render the new agent performance
80
+ agent.deploy(enable_save_replay=True)
81
+
82
+ ```
83
+ </details>
84
+
85
+ ### Run Model by Using Huggingface_ding
86
+
87
+ <details close>
88
+ <summary>(Click for Details)</summary>
89
+
90
+ ```shell
91
+ # running with trained model
92
+ python3 -u run.py
93
+ ```
94
+ **run.py**
95
+ ```python
96
+ from lzero.agent import MuZeroAgent
97
+ from huggingface_ding import pull_model_from_hub
98
+
99
+ # Pull model from Hugggingface hub
100
+ policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/PongNoFrameskip-v4-MuZero")
101
+ # Instantiate the agent
102
+ agent = MuZeroAgent(
103
+ env_id="PongNoFrameskip-v4", exp_name="PongNoFrameskip-v4-MuZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
104
+ )
105
+ # Continue training
106
+ agent.train(step=5000)
107
+ # Render the new agent performance
108
+ agent.deploy(enable_save_replay=True)
109
+
110
+ ```
111
+ </details>
112
+
113
+ ## Model Training
114
+
115
+ ### Train the Model and Push to Huggingface_hub
116
+
117
+ <details close>
118
+ <summary>(Click for Details)</summary>
119
+
120
+ ```shell
121
+ #Training Your Own Agent
122
+ python3 -u train.py
123
+ ```
124
+ **train.py**
125
+ ```python
126
+ from lzero.agent import MuZeroAgent
127
+ from huggingface_ding import push_model_to_hub
128
+
129
+ # Instantiate the agent
130
+ agent = MuZeroAgent(env_id="PongNoFrameskip-v4", exp_name="PongNoFrameskip-v4-MuZero")
131
+ # Train the agent
132
+ return_ = agent.train(step=int(500000))
133
+ # Push model to huggingface hub
134
+ push_model_to_hub(
135
+ agent=agent.best,
136
+ env_name="OpenAI/Gym/Atari",
137
+ task_name="PongNoFrameskip-v4",
138
+ algo_name="MuZero",
139
+ github_repo_url="https://github.com/opendilab/LightZero",
140
+ github_doc_model_url=None,
141
+ github_doc_env_url=None,
142
+ installation_guide='''
143
+ pip3 install DI-engine[common_env,video]
144
+ pip3 install LightZero
145
+ ''',
146
+ usage_file_by_git_clone="./muzero/pong_muzero_deploy.py",
147
+ usage_file_by_huggingface_ding="./muzero/pong_muzero_download.py",
148
+ train_file="./muzero/pong_muzero.py",
149
+ repo_id="OpenDILabCommunity/PongNoFrameskip-v4-MuZero",
150
+ platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)",
151
+ model_description="**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).",
152
+ create_repo=True
153
+ )
154
+
155
+ ```
156
+ </details>
157
+
158
+ **Configuration**
159
+ <details close>
160
+ <summary>(Click for Details)</summary>
161
+
162
+
163
+ ```python
164
+ exp_config = {
165
+ 'main_config': {
166
+ 'exp_name': 'MsPacmanNoFrameskip-v4-MuZero',
167
+ 'seed': 0,
168
+ 'env': {
169
+ 'stop_value': 1000000,
170
+ 'env_id': 'MsPacmanNoFrameskip-v4',
171
+ 'env_name': 'MsPacmanNoFrameskip-v4',
172
+ 'obs_shape': [4, 96, 96],
173
+ 'collector_env_num': 8,
174
+ 'evaluator_env_num': 3,
175
+ 'n_evaluator_episode': 3,
176
+ 'manager': {
177
+ 'shared_memory': False
178
+ }
179
+ },
180
+ 'policy': {
181
+ 'on_policy': False,
182
+ 'cuda': True,
183
+ 'multi_gpu': False,
184
+ 'bp_update_sync': True,
185
+ 'traj_len_inf': False,
186
+ 'model': {
187
+ 'observation_shape': [4, 96, 96],
188
+ 'frame_stack_num': 4,
189
+ 'action_space_size': 9,
190
+ 'downsample': True,
191
+ 'self_supervised_learning_loss': True,
192
+ 'discrete_action_encoding_type': 'one_hot',
193
+ 'norm_type': 'BN'
194
+ },
195
+ 'use_rnd_model': False,
196
+ 'sampled_algo': False,
197
+ 'gumbel_algo': False,
198
+ 'mcts_ctree': True,
199
+ 'collector_env_num': 8,
200
+ 'evaluator_env_num': 3,
201
+ 'env_type': 'not_board_games',
202
+ 'action_type': 'fixed_action_space',
203
+ 'battle_mode': 'play_with_bot_mode',
204
+ 'monitor_extra_statistics': True,
205
+ 'game_segment_length': 400,
206
+ 'transform2string': False,
207
+ 'gray_scale': False,
208
+ 'use_augmentation': True,
209
+ 'augmentation': ['shift', 'intensity'],
210
+ 'ignore_done': False,
211
+ 'update_per_collect': 1000,
212
+ 'model_update_ratio': 0.1,
213
+ 'batch_size': 256,
214
+ 'optim_type': 'SGD',
215
+ 'learning_rate': 0.2,
216
+ 'target_update_freq': 100,
217
+ 'target_update_freq_for_intrinsic_reward': 1000,
218
+ 'weight_decay': 0.0001,
219
+ 'momentum': 0.9,
220
+ 'grad_clip_value': 10,
221
+ 'n_episode': 8,
222
+ 'num_simulations': 50,
223
+ 'discount_factor': 0.997,
224
+ 'td_steps': 5,
225
+ 'num_unroll_steps': 5,
226
+ 'reward_loss_weight': 1,
227
+ 'value_loss_weight': 0.25,
228
+ 'policy_loss_weight': 1,
229
+ 'policy_entropy_loss_weight': 0,
230
+ 'ssl_loss_weight': 2,
231
+ 'lr_piecewise_constant_decay': True,
232
+ 'threshold_training_steps_for_final_lr': 50000,
233
+ 'manual_temperature_decay': False,
234
+ 'threshold_training_steps_for_final_temperature': 100000,
235
+ 'fixed_temperature_value': 0.25,
236
+ 'use_ture_chance_label_in_chance_encoder': False,
237
+ 'use_priority': True,
238
+ 'priority_prob_alpha': 0.6,
239
+ 'priority_prob_beta': 0.4,
240
+ 'root_dirichlet_alpha': 0.3,
241
+ 'root_noise_weight': 0.25,
242
+ 'random_collect_episode_num': 0,
243
+ 'eps': {
244
+ 'eps_greedy_exploration_in_collect': False,
245
+ 'type': 'linear',
246
+ 'start': 1.0,
247
+ 'end': 0.05,
248
+ 'decay': 100000
249
+ },
250
+ 'cfg_type': 'MuZeroPolicyDict',
251
+ 'reanalyze_ratio': 0.0,
252
+ 'eval_freq': 2000,
253
+ 'replay_buffer_size': 1000000
254
+ },
255
+ 'wandb_logger': {
256
+ 'gradient_logger': False,
257
+ 'video_logger': False,
258
+ 'plot_logger': False,
259
+ 'action_logger': False,
260
+ 'return_logger': False
261
+ }
262
+ },
263
+ 'create_config': {
264
+ 'env': {
265
+ 'type': 'atari_lightzero',
266
+ 'import_names': ['zoo.atari.envs.atari_lightzero_env']
267
+ },
268
+ 'env_manager': {
269
+ 'type': 'subprocess'
270
+ },
271
+ 'policy': {
272
+ 'type': 'muzero',
273
+ 'import_names': ['lzero.policy.muzero']
274
+ }
275
+ }
276
+ }
277
+
278
+ ```
279
+ </details>
280
+
281
+ **Training Procedure**
282
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
283
+ - **Weights & Biases (wandb):** [monitor link](<TODO>)
284
+
285
+ ## Model Information
286
+ <!-- Provide the basic links for the model. -->
287
+ - **Github Repository:** [repo link](https://github.com/opendilab/LightZero)
288
+ - **Doc**: [Algorithm link](<TODO>)
289
+ - **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/MsPacmanNoFrameskip-v4-MuZero/blob/main/policy_config.py)
290
+ - **Demo:** [video](https://huggingface.co/OpenDILabCommunity/MsPacmanNoFrameskip-v4-MuZero/blob/main/replay.mp4)
291
+ <!-- Provide the size information for the model. -->
292
+ - **Parameters total size:** 24020.27 KB
293
+ - **Last Update Date:** 2023-12-16
294
+
295
+ ## Environments
296
+ <!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
297
+ - **Benchmark:** OpenAI/Gym/Atari
298
+ - **Task:** MsPacmanNoFrameskip-v4
299
+ - **Gym version:** 0.25.1
300
+ - **DI-engine version:** v0.5.0
301
+ - **PyTorch version:** 2.0.1+cu117
302
+ - **Doc**: [Environments link](<TODO>)