Tora: Trajectory-oriented Diffusion Transformer for Video Generation

Zhenghao Zhang\*, Junchao Liao\*, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, Weizhi Wang \* equal contribution

## 💡 Abstract Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that integrates textual, visual, and trajectory conditions concurrently for video generation. Specifically, Tora consists of a Trajectory Extractor (TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser (MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D video compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos following trajectories. Our design aligns seamlessly with DiT’s scalability, allowing precise control of video content’s dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate Tora’s excellence in achieving high motion fidelity, while also meticulously simulating the movement of physical world. ## 📣 Updates - `2024/10/31` Model weights uploaded to [HuggingFace](https://huggingface.co./Le0jc/Tora). We also provided an English demo on [ModelScope](https://www.modelscope.cn/studios/Alibaba_Research_Intelligence_Computing/Tora_En). - `2024/10/23` 🔥🔥Our [ModelScope Demo](https://www.modelscope.cn/studios/xiaoche/Tora) is launched. Welcome to try it out! We also upload the model weights to [ModelScope](https://www.modelscope.cn/models/xiaoche/Tora). - `2024/10/21` Thanks to [@kijai](https://github.com/kijai) for supporting Tora in ComfyUI! [Link](https://github.com/kijai/ComfyUI-CogVideoXWrapper) - `2024/10/15` 🔥🔥We released our inference code and model weights. **Please note that this is a CogVideoX version of Tora, built on the CogVideoX-5B model. This version of Tora is meant for academic research purposes only. Due to our commercial plans, we will not be open-sourcing the complete version of Tora at this time.** - `2024/08/27` We released our v2 paper including appendix. - `2024/07/31` We submitted our paper on arXiv and released our project page. ## 📑 Table of Contents - [Showcases](#%EF%B8%8F-showcases) - [Model Weights](#-model-weights) - [Inference](#-inference) - [Acknowledgements](#-acknowledgements) - [Our previous work](#-our-previous-work) - [Citation](#-citation) ## 🎞️ Showcases https://github.com/user-attachments/assets/949d5e99-18c9-49d6-b669-9003ccd44bf1 https://github.com/user-attachments/assets/7e7dbe87-a8ba-4710-afd0-9ef528ec329b https://github.com/user-attachments/assets/4026c23d-229d-45d7-b5be-6f3eb9e4fd50 All videos are available in this [Link](https://cloudbook-public-daily.oss-cn-hangzhou.aliyuncs.com/Tora_t2v/showcases.zip) ## 📦 Model Weights ### Folder Structure ``` Tora └── sat └── ckpts ├── t5-v1_1-xxl │ ├── model-00001-of-00002.safetensors │ └── ... ├── vae │ └── 3d-vae.pt └── tora └── t2v └── mp_rank_00_model_states.pt ``` ### Download Links *Note: Downloading the `tora` weights requires following the [CogVideoX License](CogVideoX_LICENSE).* You can choose one of the following options: HuggingFace, ModelScope, or native links. After downloading the model weights, you can put them in the `Tora/sat/ckpts` folder. #### HuggingFace ```bash # This can be faster pip install "huggingface_hub[hf_transfer]" HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download Le0jc/Tora --local-dir ckpts ``` or ```bash # use git git lfs install git clone https://huggingface.co./Le0jc/Tora ``` #### ModelScope - SDK ```bash from modelscope import snapshot_download model_dir = snapshot_download('xiaoche/Tora') ``` - Git ```bash git clone https://www.modelscope.cn/xiaoche/Tora.git ``` #### Native - Download the VAE and T5 model following [CogVideo](https://github.com/THUDM/CogVideo/blob/main/sat/README.md#2-download-model-weights): - VAE: https://cloud.tsinghua.edu.cn/f/fdba7608a49c463ba754/?dl=1 - T5: [text_encoder](https://huggingface.co./THUDM/CogVideoX-2b/tree/main/text_encoder), [tokenizer](https://huggingface.co./THUDM/CogVideoX-2b/tree/main/tokenizer) - Tora t2v model weights: [Link](https://cloudbook-public-daily.oss-cn-hangzhou.aliyuncs.com/Tora_t2v/mp_rank_00_model_states.pt). Downloading this weight requires following the [CogVideoX License](CogVideoX_LICENSE). ## 🔄 Inference please refer to our [Github](https://github.com/alibaba/Tora) or [modelscope online demo](https://www.modelscope.cn/studios/Alibaba_Research_Intelligence_Computing/Tora_En) ### Recommendations for Text Prompts For text prompts, we highly recommend using GPT-4 to enhance the details. Simple prompts may negatively impact both visual quality and motion control effectiveness. You can refer to the following resources for guidance: - [CogVideoX Documentation](https://github.com/THUDM/CogVideo/blob/main/inference/convert_demo.py) - [OpenSora Scripts](https://github.com/hpcaitech/Open-Sora/blob/main/scripts/inference.py) ## 🤝 Acknowledgements We would like to express our gratitude to the following open-source projects that have been instrumental in the development of our project: - [CogVideo](https://github.com/THUDM/CogVideo): An open source video generation framework by THUKEG. - [Open-Sora](https://github.com/hpcaitech/Open-Sora): An open source video generation framework by HPC-AI Tech. - [MotionCtrl](https://github.com/TencentARC/MotionCtrl): A video generation model supporting motion control by ARC Lab, Tencent PCG. - [ComfyUI-DragNUWA](https://github.com/chaojie/ComfyUI-DragNUWA): An implementation of DragNUWA for ComfyUI. Special thanks to the contributors of these libraries for their hard work and dedication! ## 📄 Our previous work - [AnimateAnything: Fine Grained Open Domain Image Animation with Motion Guidance](https://github.com/alibaba/animate-anything) ## 📚 Citation ```bibtex @misc{zhang2024toratrajectoryorienteddiffusiontransformer, title={Tora: Trajectory-oriented Diffusion Transformer for Video Generation}, author={Zhenghao Zhang and Junchao Liao and Menghao Li and Zuozhuo Dai and Bingxue Qiu and Siyu Zhu and Long Qin and Weizhi Wang}, year={2024}, eprint={2407.21705}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2407.21705}, } ```