| ## Environment Setup | |
| `pip install -r requirements.txt` | |
| ## Download checkpoints | |
| 1. Download the pretrained checkpoints of [SVD_xt](https://huggingface.co./stabilityai/stable-video-diffusion-img2vid-xt-1-1) from huggingface to `./ckpts`. | |
| 2. Download the checkpint of [MOFA-Adapter](https://huggingface.co./MyNiuuu/MOFA-Video-Traj) from huggingface to `./ckpts`. | |
| The final structure of checkpoints should be: | |
| ```text | |
| ./ckpts/ | |
| |-- controlnet | |
| | |-- config.json | |
| | `-- diffusion_pytorch_model.safetensors | |
| |-- stable-video-diffusion-img2vid-xt-1-1 | |
| | |-- feature_extractor | |
| | |-- ... | |
| | |-- image_encoder | |
| | |-- ... | |
| | |-- scheduler | |
| | |-- ... | |
| | |-- unet | |
| | |-- ... | |
| | |-- unet_ch9 | |
| | |-- ... | |
| | |-- vae | |
| | |-- ... | |
| | |-- svd_xt_1_1.safetensors | |
| | `-- model_index.json | |
| ``` | |
| ## Run Gradio Demo | |
| `python run_gradio.py` | |
| Please refer to the instructions on the gradio interface during the inference process. | |
| ## Paper | |
| arxiv.org/abs/2405.20222 |