Update Config

README.md

@@ -14,7 +14,32 @@ EasyAnimate是一个基于transformer结构的pipeline，可用于生成AI图片
 [English](./README_en.md) | [简体中文](./README.md)
 
 # 模型地址
-EasyAnimateV5.1:
+EasyAnimateV5.1 for diffusers:
+
+该权重文件主要用于 **[diffusers](https://github.com/huggingface/diffusers)** 仓库。  
+需要注意的是，**EasyAnimate** 仓库与 **diffusers** 仓库在权重格式和使用方式上存在些许差异，请务必仔细甄别。
+
+7B:
+| 名称 | 种类 | 存储空间 | Hugging Face | 描述 |
+|--|--|--|--|--|
+| EasyAnimateV5.1-7b-zh-InP | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers) | 官方的图生视频权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+| EasyAnimateV5.1-7b-zh-Control | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-Control-diffusers) | 官方的视频控制权重，支持不同的控制条件，如Canny、Depth、Pose、MLSD等，同时支持使用轨迹控制。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+| EasyAnimateV5.1-7b-zh-Control-Camera | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-Control-Camera-diffusers) | 官方的视频相机控制权重，支持通过输入相机运动轨迹控制生成方向。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+| EasyAnimateV5.1-7b-zh | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-diffusers) | 官方的文生视频权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+
+12B:
+| 名称 | 种类 | 存储空间 | Hugging Face | 描述 |
+|--|--|--|--|--|
+| EasyAnimateV5.1-12b-zh-InP | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers) | 官方的图生视频权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+| EasyAnimateV5.1-12b-zh-Control | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers) | 官方的视频控制权重，支持不同的控制条件，如Canny、Depth、Pose、MLSD等，同时支持使用轨迹控制。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+| EasyAnimateV5.1-12b-zh-Control-Camera | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera-diffusers) | 官方的视频相机控制权重，支持通过输入相机运动轨迹控制生成方向。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+| EasyAnimateV5.1-12b-zh | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-diffusers) | 官方的文生视频权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
+
+<details>
+  <summary>EasyAnimateV5.1:</summary>
+
+该权重文件主要用于 **[EasyAnimate](https://github.com/aigc-apps/EasyAnimate)** 仓库。  
+需要注意的是，**EasyAnimate** 仓库与 **diffusers** 仓库在权重格式和使用方式上存在些许差异，请务必仔细甄别。
 
 7B:
 | 名称 | 种类 | 存储空间 | Hugging Face | Model Scope | 描述 |
@@ -32,6 +57,7 @@ EasyAnimateV5.1:
 | EasyAnimateV5.1-12b-zh-Control-Camera | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera) | [😄Link](https://modelscope.cn/models/PAI/EasyAnimateV5.1-12b-zh-Control-Camera)| 官方的视频相机控制权重，支持通过输入相机运动轨迹控制生成方向。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
 | EasyAnimateV5.1-12b-zh | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh) | [😄Link](https://modelscope.cn/models/PAI/EasyAnimateV5.1-12b-zh)| 官方的文生视频权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以49帧、每秒8帧进行训练，支持多语言预测 |
 
+</details>
 
 # 视频作品
 
@@ -213,116 +239,436 @@ EasyAnimateV5.1:
 
 # 如何使用
 
-#### a、显存节省方案
-由于EasyAnimateV5和V5.1的参数非常大，我们需要考虑显存节省方案，以节省显存适应消费级显卡。我们给每个预测文件都提供了GPU_memory_mode，可以在model_cpu_offload，model_cpu_offload_and_qfloat8，sequential_cpu_offload中进行选择。
-
-- model_cpu_offload代表整个模型在使用后会进入cpu，可以节省部分显存。
-- model_cpu_offload_and_qfloat8代表整个模型在使用后会进入cpu，并且对transformer模型进行了float8的量化，可以节省更多的显存。
-- sequential_cpu_offload代表模型的每一层在使用后会进入cpu，速度较慢，节省大量显存。
-
-qfloat8会降低模型的性能，但可以节省更多的显存。如果显存足够，推荐使用model_cpu_offload。
-
-#### b、通过comfyui
-具体查看[ComfyUI README](https://github.com/aigc-apps/EasyAnimate/blob/main/comfyui/README.md)。
-
-#### c、运行python文件
-- 步骤1：下载对应[权重](#model-zoo)放入models文件夹。
-- 步骤2：根据不同的权重与预测目标使用不同的文件进行预测。
-  - 文生视频：
-    - 使用predict_t2v.py文件中修改prompt、neg_prompt、guidance_scale和seed。
-    - 而后运行predict_t2v.py文件，等待生成结果，结果保存在samples/easyanimate-videos文件夹中。
-  - 图生视频：
-    - 使用predict_i2v.py文件中修改validation_image_start、validation_image_end、prompt、neg_prompt、guidance_scale和seed。
-    - validation_image_start是视频的开始图片，validation_image_end是视频的结尾图片。
-    - 而后运行predict_i2v.py文件，等待生成结果，结果保存在samples/easyanimate-videos_i2v文件夹中。
-  - 视频生视频：
-    - 使用predict_v2v.py文件中修改validation_video、validation_image_end、prompt、neg_prompt、guidance_scale和seed。
-    - validation_video是视频生视频的参考视频。您可以使用以下视频运行演示：[演示视频](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/play_guitar.mp4)
-    - 而后运行predict_v2v.py文件，等待生成结果，结果保存在samples/easyanimate-videos_v2v文件夹中。
-  - 普通控制生视频（Canny、Pose、Depth等）：
-    - 使用predict_v2v_control.py文件中修改control_video、validation_image_end、prompt、neg_prompt、guidance_scale和seed。
-    - control_video是控制生视频的控制视频，是使用Canny、Pose、Depth等算子提取后的视频。您可以使用以下视频运行演示：[演示视频](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1.1/pose.mp4)
-    - 而后运行predict_v2v_control.py文件，等待生成结果，结果保存在samples/easyanimate-videos_v2v_control文件夹中。
-  - 轨迹控制视频：
-    - 使用predict_v2v_control.py文件中修改control_video、ref_image、validation_image_end、prompt、neg_prompt、guidance_scale和seed。
-    - control_video是轨迹控制视频的控制视频，ref_image是参考的首帧图片。您可以使用以下图片和控制视频运行演示：[演示图像](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v5.1/dog.png)，[演示视频](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v5.1/trajectory_demo.mp4)
-    - 而后运行predict_v2v_control.py文件，等待生成结果，结果保存在samples/easyanimate-videos_v2v_control文件夹中。
-    - 推荐使用ComfyUI进行交互。
-  - 相机控制视频：
-    - 使用predict_v2v_control.py文件中修改control_video、ref_image、validation_image_end、prompt、neg_prompt、guidance_scale和seed。
-    - control_camera_txt是相机控制视频的控制文件，ref_image是参考的首帧图片。您可以使用以下图片和控制视频运行演示：[演示图像](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/firework.png)，[演示文件（来自于CameraCtrl）](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v5.1/0a3b5fb184936a83.txt)
-    - 而后运行predict_v2v_control.py文件，等待生成结果，结果保存在samples/easyanimate-videos_v2v_control文件夹中。
-    - 推荐使用ComfyUI进行交互。
-- 步骤3：如果想结合自己训练的其他backbone与Lora，则看情况修改predict_t2v.py中的predict_t2v.py和lora_path。
-
-#### d、通过ui界面
-
-webui支持文生视频、图生视频、视频生视频和普通控制生视频（Canny、Pose、Depth等）
-
-- 步骤1：下载对应[权重](#model-zoo)放入models文件夹。
-- 步骤2：运行app.py文件，进入gradio页面。
-- 步骤3：根据页面选择生成模型，填入prompt、neg_prompt、guidance_scale和seed等，点击生成，等待生成结果，结果保存在sample文件夹中。
-
-# 快速启动
-### 1. 云使用: AliyunDSW/Docker
-#### a. 通过阿里云 DSW
-DSW 有免费 GPU 时间，用户可申请一次，申请后3个月内有效。
-
-阿里云在[Freetier](https://free.aliyun.com/?product=9602825&crowd=enterprise&spm=5176.28055625.J_5831864660.1.e939154aRgha4e&scm=20140722.M_9974135.P_110.MO_1806-ID_9974135-MID_9974135-CID_30683-ST_8512-V_1)提供免费GPU时间，获取并在阿里云PAI-DSW中使用，5分钟内即可启动EasyAnimate
-
-[![DSW Notebook](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/dsw.png)](https://gallery.pai-ml.com/#/preview/deepLearning/cv/easyanimate_v5)
-
-#### b. 通过ComfyUI
-我们的ComfyUI界面如下，具体查看[ComfyUI README](https://github.com/aigc-apps/EasyAnimate/blob/main/comfyui/README.md)。
-![workflow graph](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v3/comfyui_i2v.jpg)
-
-#### c. 通过docker
-使用docker的情况下，请保证机器中已经正确安装显卡驱动与CUDA环境，然后以此执行以下命令：
+#### a、Text to video
+```python
+import torch
+import numpy as np
+from diffusers import EasyAnimatePipeline
+from diffusers.utils import export_to_video
+
+# Models: "alibaba-pai/EasyAnimateV5.1-7b-zh-diffusers" or "alibaba-pai/EasyAnimateV5.1-12b-zh-diffusers"
+pipe = EasyAnimatePipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = (
+    "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. "
+    "The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other "
+    "pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, "
+    "casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. "
+    "The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical "
+    "atmosphere of this unique musical performance."
+)
+negative_prompt     = "bad detailed"
+height              = 512
+width               = 512
+guidance_scale      = 6
+num_inference_steps = 50
+num_frames          = 49
+seed                = 43
+generator           = torch.Generator(device="cuda").manual_seed(seed)
+
+video = pipe(
+    prompt=prompt, 
+    negative_prompt=negative_prompt,
+    guidance_scale=guidance_scale, 
+    num_inference_steps=num_inference_steps,
+    num_frames=num_frames,
+    height=height,
+    width=width,
+    generator=generator,
+).frames[0]
+export_to_video(video, "output.mp4", fps=8)
 ```
-# pull image
-docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate
-
-# enter image
-docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate
-
-# clone code
-git clone https://github.com/aigc-apps/EasyAnimate.git
 
-# enter EasyAnimate's dir
-cd EasyAnimate
-
-# download weights
-mkdir models/Diffusion_Transformer
-mkdir models/Motion_Module
-mkdir models/Personalized_Model
-
-# Please use the hugginface link or modelscope link to download the EasyAnimateV5.1 model.
-# https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP
-# https://modelscope.cn/models/PAI/EasyAnimateV5.1-12b-zh-InP
+#### b、Image to video
+```python
+import torch
+from diffusers import EasyAnimateInpaintPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.utils import export_to_video, load_image, load_video
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers"
+pipe = EasyAnimateInpaintPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+
+validation_image_start = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
+validation_image_end = None
+sample_size = (448, 576)
+num_frames = 49
+
+input_video, input_video_mask = get_image_to_video_latent([validation_image_start], validation_image_end, num_frames, sample_size)
+
+video = pipe(
+    prompt, 
+    negative_prompt=negative_prompt,
+    num_frames=num_frames, 
+    height=sample_size[0], 
+    width=sample_size[1], 
+    video=input_video, 
+    mask_video=input_video_mask
+)
+export_to_video(video.frames[0], "output.mp4", fps=8)
+```
+#### c、Video to video
+```python
+import torch
+from diffusers import EasyAnimateInpaintPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.utils import export_to_video, load_image, load_video
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers"
+pipe = EasyAnimateInpaintPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = "一只穿着小外套的猫咪正安静地坐在花园的秋千上弹吉他。它的小外套精致而合身，增添了几分俏皮与可爱。晚霞的余光洒在它柔软的毛皮上，给它的毛发镀上了一层温暖的金色光辉。和煦的微风轻轻拂过，带来阵阵花香和草木的气息，令人心旷神怡。周围斑驳的光影随着音乐的旋律轻轻摇曳，仿佛整个花园都在为这只小猫咪的演奏伴舞。阳光透过树叶间的缝隙，投下一片片光影交错的图案，与悠扬的吉他声交织在一起，营造出一种梦幻而宁静的氛围。猫咪专注而投入地弹奏着，每一个音符都似乎充满了魔力，让这个傍晚变得更加美好。"
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+sample_size = (384, 672)
+num_frames = 49
+input_video = load_video("https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP/resolve/main/asset/1.mp4")
+input_video, input_video_mask, _ = get_video_to_video_latent(input_video, num_frames=num_frames, validation_video_mask=None, sample_size=sample_size)
+video = pipe(
+    prompt, 
+    num_frames=num_frames, 
+    negative_prompt=negative_prompt,
+    height=sample_size[0], 
+    width=sample_size[1], 
+    video=input_video, 
+    mask_video=input_video_mask,
+    strength=0.70
+)
+
+export_to_video(video.frames[0], "output.mp4", fps=8)
 ```
 
-### 2. 本地安装: 环境检查/下载/安装
-#### a. 环境检查
-我们已验证EasyAnimate可在以下环境中执行：
-
-Windows 的详细信息：
-- 操作系统 Windows 10
-- python: python3.10 & python3.11
-- pytorch: torch2.2.0
-- CUDA: 11.8 & 12.1
-- CUDNN: 8+
-- GPU： Nvidia-3060 12G
-
-Linux 的详细信息：
-- 操作系统 Ubuntu 20.04, CentOS
-- python: python3.10 & python3.11
-- pytorch: torch2.2.0
-- CUDA: 11.8 & 12.1
-- CUDNN: 8+
-- GPU：Nvidia-V100 16G & Nvidia-A10 24G & Nvidia-A100 40G & Nvidia-A100 80G
+#### d、Control to video
+```python
+import numpy as np
+import torch
+from diffusers import EasyAnimateControlPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.utils import export_to_video, load_video
+from PIL import Image
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-Control-diffusers"
+pipe = EasyAnimateControlPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+control_video = load_video(
+    "https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control/resolve/main/asset/pose.mp4"
+)
+prompt = (
+    "In this sunlit outdoor garden, a beautiful woman is dressed in a knee-length, sleeveless white dress. "
+    "The hem of her dress gently sways with her graceful dance, much like a butterfly fluttering in the breeze. "
+    "Sunlight filters through the leaves, casting dappled shadows that highlight her soft features and clear eyes, "
+    "making her appear exceptionally elegant. It seems as if every movement she makes speaks of youth and vitality. "
+    "As she twirls on the grass, her dress flutters, as if the entire garden is rejoicing in her dance. "
+    "The colorful flowers around her sway in the gentle breeze, with roses, chrysanthemums, and lilies each "
+    "releasing their fragrances, creating a relaxed and joyful atmosphere."
+)
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+sample_size = (672, 384)
+num_frames = 49
+generator = torch.Generator(device="cuda").manual_seed(43)
+input_video, _, _ = get_video_to_video_latent(np.array(control_video), num_frames, sample_size)
+
+video = pipe(prompt, num_frames=num_frames, negative_prompt=negative_prompt, height=sample_size[0], width=sample_size[1], control_video=input_video, generator=generator).frames[0]
+export_to_video(video, "output.mp4", fps=8)
+```
 
-我们需要大约 60GB 的可用磁盘空间，请检查！
+#### e、Camera Control to video
+由于镜头控制模型需要对镜头文件进行处理，请前往[asset](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera/blob/main/asset/)下载对应镜头移动的txt文件。
+
+相关代码较为复杂，已经隐藏，请点击打开。
+<details>
+  <summary>EasyAnimateV5.1:</summary>
+
+```python
+import numpy as np
+import torch
+from diffusers import EasyAnimateControlPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.utils import export_to_video, load_video, load_image
+from einops import rearrange
+from packaging import version as pver
+from PIL import Image
+
+
+class Camera(object):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    def __init__(self, entry):
+        fx, fy, cx, cy = entry[1:5]
+        self.fx = fx
+        self.fy = fy
+        self.cx = cx
+        self.cy = cy
+        w2c_mat = np.array(entry[7:]).reshape(3, 4)
+        w2c_mat_4x4 = np.eye(4)
+        w2c_mat_4x4[:3, :] = w2c_mat
+        self.w2c_mat = w2c_mat_4x4
+        self.c2w_mat = np.linalg.inv(w2c_mat_4x4)
+
+def custom_meshgrid(*args):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    # ref: https://pytorch.org/docs/stable/generated/torch.meshgrid.html?highlight=meshgrid#torch.meshgrid
+    if pver.parse(torch.__version__) < pver.parse('1.10'):
+        return torch.meshgrid(*args)
+    else:
+        return torch.meshgrid(*args, indexing='ij')
+
+def get_relative_pose(cam_params):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    abs_w2cs = [cam_param.w2c_mat for cam_param in cam_params]
+    abs_c2ws = [cam_param.c2w_mat for cam_param in cam_params]
+    cam_to_origin = 0
+    target_cam_c2w = np.array([
+        [1, 0, 0, 0],
+        [0, 1, 0, -cam_to_origin],
+        [0, 0, 1, 0],
+        [0, 0, 0, 1]
+    ])
+    abs2rel = target_cam_c2w @ abs_w2cs[0]
+    ret_poses = [target_cam_c2w, ] + [abs2rel @ abs_c2w for abs_c2w in abs_c2ws[1:]]
+    ret_poses = np.array(ret_poses, dtype=np.float32)
+    return ret_poses
+
+def ray_condition(K, c2w, H, W, device):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    # c2w: B, V, 4, 4
+    # K: B, V, 4
+
+    B = K.shape[0]
+
+    j, i = custom_meshgrid(
+        torch.linspace(0, H - 1, H, device=device, dtype=c2w.dtype),
+        torch.linspace(0, W - 1, W, device=device, dtype=c2w.dtype),
+    )
+    i = i.reshape([1, 1, H * W]).expand([B, 1, H * W]) + 0.5  # [B, HxW]
+    j = j.reshape([1, 1, H * W]).expand([B, 1, H * W]) + 0.5  # [B, HxW]
+
+    fx, fy, cx, cy = K.chunk(4, dim=-1)  # B,V, 1
+
+    zs = torch.ones_like(i)  # [B, HxW]
+    xs = (i - cx) / fx * zs
+    ys = (j - cy) / fy * zs
+    zs = zs.expand_as(ys)
+
+    directions = torch.stack((xs, ys, zs), dim=-1)  # B, V, HW, 3
+    directions = directions / directions.norm(dim=-1, keepdim=True)  # B, V, HW, 3
+
+    rays_d = directions @ c2w[..., :3, :3].transpose(-1, -2)  # B, V, 3, HW
+    rays_o = c2w[..., :3, 3]  # B, V, 3
+    rays_o = rays_o[:, :, None].expand_as(rays_d)  # B, V, 3, HW
+    # c2w @ dirctions
+    rays_dxo = torch.cross(rays_o, rays_d)
+    plucker = torch.cat([rays_dxo, rays_d], dim=-1)
+    plucker = plucker.reshape(B, c2w.shape[1], H, W, 6)  # B, V, H, W, 6
+    # plucker = plucker.permute(0, 1, 4, 2, 3)
+    return plucker
+
+def process_pose_file(pose_file_path, width=672, height=384, original_pose_width=1280, original_pose_height=720, device='cpu', return_poses=False):
+    """Modified from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    with open(pose_file_path, 'r') as f:
+        poses = f.readlines()
+
+    poses = [pose.strip().split(' ') for pose in poses[1:]]
+    cam_params = [[float(x) for x in pose] for pose in poses]
+    if return_poses:
+        return cam_params
+    else:
+        cam_params = [Camera(cam_param) for cam_param in cam_params]
+
+        sample_wh_ratio = width / height
+        pose_wh_ratio = original_pose_width / original_pose_height  # Assuming placeholder ratios, change as needed
+
+        if pose_wh_ratio > sample_wh_ratio:
+            resized_ori_w = height * pose_wh_ratio
+            for cam_param in cam_params:
+                cam_param.fx = resized_ori_w * cam_param.fx / width
+        else:
+            resized_ori_h = width / pose_wh_ratio
+            for cam_param in cam_params:
+                cam_param.fy = resized_ori_h * cam_param.fy / height
+
+        intrinsic = np.asarray([[cam_param.fx * width,
+                                cam_param.fy * height,
+                                cam_param.cx * width,
+                                cam_param.cy * height]
+                                for cam_param in cam_params], dtype=np.float32)
+
+        K = torch.as_tensor(intrinsic)[None]  # [1, 1, 4]
+        c2ws = get_relative_pose(cam_params)  # Assuming this function is defined elsewhere
+        c2ws = torch.as_tensor(c2ws)[None]  # [1, n_frame, 4, 4]
+        plucker_embedding = ray_condition(K, c2ws, height, width, device=device)[0].permute(0, 3, 1, 2).contiguous()  # V, 6, H, W
+        plucker_embedding = plucker_embedding[None]
+        plucker_embedding = rearrange(plucker_embedding, "b f c h w -> b f h w c")[0]
+        return plucker_embedding
+    
+def get_image_latent(ref_image=None, sample_size=None):
+    if ref_image is not None:
+        if isinstance(ref_image, str):
+            ref_image = Image.open(ref_image).convert("RGB")
+            ref_image = ref_image.resize((sample_size[1], sample_size[0]))
+            ref_image = torch.from_numpy(np.array(ref_image))
+            ref_image = ref_image.unsqueeze(0).permute([3, 0, 1, 2]).unsqueeze(0) / 255
+        else:
+            ref_image = torch.from_numpy(np.array(ref_image))
+            ref_image = ref_image.unsqueeze(0).permute([3, 0, 1, 2]).unsqueeze(0) / 255
+
+    return ref_image
+
+# Models: "alibaba-pai/EasyAnimateV5.1-7b-zh-Control-Camera-diffusers" or "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera-diffusers"
+pipe = EasyAnimateControlPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+input_video, input_video_mask = None, None
+prompt = "Fireworks light up the evening sky over a sprawling cityscape with gothic-style buildings featuring pointed towers and clock faces. The city is lit by both artificial lights from the buildings and the colorful bursts of the fireworks. The scene is viewed from an elevated angle, showcasing a vibrant urban environment set against a backdrop of a dramatic, partially cloudy sky at dusk."
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+sample_size = (384, 672)
+num_frames = 49
+fps = 8
+ref_image = load_image("https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera/resolve/main/asset/1.png")
+
+control_camera_video = process_pose_file("/The_Path_To/Pan_Left.txt", sample_size[1], sample_size[0])
+control_camera_video = control_camera_video[::int(24 // fps)][:num_frames].permute([3, 0, 1, 2]).unsqueeze(0)
+ref_image = get_image_latent(sample_size=sample_size, ref_image=ref_image)
+video = pipe(
+    prompt, 
+    negative_prompt=negative_prompt, 
+    num_frames=num_frames, 
+    height=sample_size[0], 
+    width=sample_size[1], 
+    control_camera_video=control_camera_video, 
+    ref_image=ref_image
+).frames[0]
+export_to_video(video, "output.mp4", fps=fps)
+```
+</details>
+
+#### f、float8 model
+由于EasyAnimateV5.1的参数非常大，我们需要考虑显存节省方案，以节省显存适应消费级显卡。我们可以将模型放入float8以节省显存。
+
+以图生视频为例，我们将模型防止到float8上，然后在预测的时候转为bfloat16。
+```python
+"""Modified from https://github.com/kijai/ComfyUI-MochiWrapper
+"""
+import torch
+import torch.nn as nn
+from diffusers import EasyAnimateInpaintPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.utils import export_to_video, load_image, load_video
+
+def autocast_model_forward(cls, origin_dtype, *inputs, **kwargs):
+    weight_dtype = cls.weight.dtype
+    cls.to(origin_dtype)
+
+    # Convert all inputs to the original dtype
+    inputs = [input.to(origin_dtype) for input in inputs]
+    out = cls.original_forward(*inputs, **kwargs)
+
+    cls.to(weight_dtype)
+    return out
+
+def convert_weight_dtype_wrapper(module, origin_dtype):
+    for name, module in module.named_modules():
+        if name == "" or "embed_tokens" in name:
+            continue
+        original_forward = module.forward
+        if hasattr(module, "weight"):
+            setattr(module, "original_forward", original_forward)
+            setattr(
+                module,
+                "forward",
+                lambda *inputs, m=module, **kwargs: autocast_model_forward(m, origin_dtype, *inputs, **kwargs)
+            )
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers"
+pipe = EasyAnimateInpaintPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.transformer = pipe.transformer.to(torch.float8_e4m3fn)
+from fp8_optimization import convert_weight_dtype_wrapper
+
+for _text_encoder in [pipe.text_encoder, pipe.text_encoder_2]:
+    if hasattr(_text_encoder, "visual"):
+        del _text_encoder.visual
+convert_weight_dtype_wrapper(pipe.transformer, torch.bfloat16)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+validation_image_start = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
+validation_image_end = None
+sample_size = (448, 576)
+num_frames = 49
+input_video, input_video_mask = get_image_to_video_latent(
+    [validation_image_start], validation_image_end, num_frames, sample_size
+)
+
+video = pipe(
+    prompt, 
+    negative_prompt=negative_prompt,
+    num_frames=num_frames, 
+    height=sample_size[0], 
+    width=sample_size[1], 
+    video=input_video, 
+    mask_video=input_video_mask
+)
+export_to_video(video.frames[0], "output.mp4", fps=8)
+```
 
+# 显存需求
 EasyAnimateV5.1-12B的视频大小可以由不同的GPU Memory生成，包括：
 | GPU memory |384x672x25|384x672x49|576x1008x25|576x1008x49|768x1344x25|768x1344x49|
 |----------|----------|----------|----------|----------|----------|----------|
@@ -339,9 +685,9 @@ EasyAnimateV5.1-7B的视频大小可以由不同的GPU Memory生成，包括：
 | 40GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 
 | 80GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 
 
-✅ 表示它可以在"model_cpu_offload"的情况下运行，🧡代表它可以在"model_cpu_offload_and_qfloat8"的情况下运行，⭕️ 表示它可以在"sequential_cpu_offload"的情况下运行，❌ 表示它无法运行。请注意，使用sequential_cpu_offload运行会更慢。
+✅ 表示它可以在"model_cpu_offload"的情况下运行，🧡代表它可以在"model_cpu_offload" + float8的情况下运行，⭕️ 表示它可以在"sequential_cpu_offload"的情况下运行，当前由于qwen2vl不支持"sequential_cpu_offload"，我们无法使用在diffusers中使用sequential_cpu_offload，具体修复时间未知，但可以在EasyAnimate官方仓库中使用，❌ 表示它无法运行。请注意，使用sequential_cpu_offload运行会更慢。
 
-有一些不支持torch.bfloat16的卡型，如2080ti、V100，需要将app.py、predict文件中的weight_dtype修改为torch.float16才可以运行。
+有一些不支持torch.bfloat16的卡型，如2080ti、V100，需要使用torch.float16才可以运行。
 
 EasyAnimateV5.1-12B使用不同GPU在25个steps中的生成时间如下：
 | GPU |384x672x72|384x672x49|576x1008x25|576x1008x49|768x1344x25|768x1344x49|
@@ -349,29 +695,6 @@ EasyAnimateV5.1-12B使用不同GPU在25个steps中的生成时间如下：
 | A10 24GB |约120秒 (4.8s/it)|约240秒 (9.6s/it)|约320秒 (12.7s/it)| 约750秒 (29.8s/it)| ❌ | ❌ |
 | A100 80GB |约45秒 (1.75s/it)|约90秒 (3.7s/it)|约120秒 (4.7s/it)|约300秒 (11.4s/it)|约265秒 (10.6s/it)| 约710秒 (28.3s/it)|
 
-
-#### b. 权重放置
-我们最好将[权重](#model-zoo)按照指定路径进行放置：
-
-EasyAnimateV5.1:
-```
-📦 models/
-├── 📂 Diffusion_Transformer/
-│   ├── 📂 EasyAnimateV5.1-12b-zh-InP/
-│   ├── 📂 EasyAnimateV5.1-12b-zh-Control/
-│   ├── 📂 EasyAnimateV5.1-12b-zh-Control-Camera/
-│   └── 📂 EasyAnimateV5.1-12b-zh/
-├── 📂 Personalized_Model/
-│   └── your trained trainformer model / your trained lora model (for UI load)
-```
-
-# 联系我们
-1. 扫描下方二维码或搜索群号：77450006752 来加入钉钉群。
-2. 扫描下方二维码来加入微信群（如果二维码失效，可扫描最右边同学的微信，邀请您入群）
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/dd.png" alt="ding group" width="30%"/>
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/wechat.jpg" alt="Wechat group" width="30%"/>
-<img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/person.jpg" alt="Person" width="30%"/>
-
 # 参考文献
 - CogVideo: https://github.com/THUDM/CogVideo/
 - Flux: https://github.com/black-forest-labs/flux

README_en.md

@@ -13,7 +13,30 @@ EasyAnimate is a pipeline based on the transformer architecture, designed for ge
 
 # Model zoo
 
-EasyAnimateV5.1:
+EasyAnimateV5.1 for diffusers:
+
+This weight file is mainly used for the **[diffusers](https://github.com/huggingface/diffusers)**  repository.
+
+Please note that there are some differences in the weight format and usage between the EasyAnimate repository and the diffusers repository. Be sure to carefully distinguish between them.
+
+7B:
+| Name | Type | Storage Space | Hugging Face | Description |
+|--|--|--|--|--|
+| EasyAnimateV5.1-7b-zh-InP | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers) | Official image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+| EasyAnimateV5.1-7b-zh-Control | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-Control-diffusers) | Official video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, and trajectory control. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+| EasyAnimateV5.1-7b-zh-Control-Camera | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-Control-Camera-diffusers) | Official video camera control weights, supporting direction generation control by inputting camera motion trajectories. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+| EasyAnimateV5.1-7b-zh | EasyAnimateV5.1 | 30 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-7b-zh-diffusers) | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+
+12B:
+| Name | Type | Storage Space | Hugging Face | Description |
+|--|--|--|--|--|
+| EasyAnimateV5.1-12b-zh-InP | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers) | Official image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+| EasyAnimateV5.1-12b-zh-Control | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers) | Official video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, and trajectory control. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+| EasyAnimateV5.1-12b-zh-Control-Camera | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera-diffusers) | Official video camera control weights, supporting direction generation control by inputting camera motion trajectories. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+| EasyAnimateV5.1-12b-zh | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-diffusers) | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
+
+<details>
+  <summary>EasyAnimateV5.1:</summary>
 
 7B:
 | Name | Type | Storage Space | Hugging Face | Model Scope | Description |
@@ -31,6 +54,8 @@ EasyAnimateV5.1:
 | EasyAnimateV5.1-12b-zh-Control-Camera | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera) | [😄Link](https://modelscope.cn/models/PAI/EasyAnimateV5.1-12b-zh-Control-Camera) | Official video camera control weights, supporting direction generation control by inputting camera motion trajectories. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
 | EasyAnimateV5.1-12b-zh | EasyAnimateV5.1 | 39 GB | [🤗Link](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh) | [😄Link](https://modelscope.cn/models/PAI/EasyAnimateV5.1-12b-zh) | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports for multilingual prediction. |
 
+</details>
+
 # Video Result
 
 ### Image to Video with EasyAnimateV5.1-12b-zh-InP
@@ -212,118 +237,437 @@ Generic Control Video (Canny, Pose, Depth, etc.):
 
 # How to use
 
-#### a. Memory-Saving Options
-Since EasyAnimateV5 and V5.1 have very large parameters, we need to consider memory-saving options to adapt to consumer-grade graphics cards. We provide GPU_memory_mode for each prediction file, allowing you to choose from model_cpu_offload, model_cpu_offload_and_qfloat8, or sequential_cpu_offload.
-
-- model_cpu_offload means the entire model will move to the CPU after use, saving some memory.
-- model_cpu_offload_and_qfloat8 means the entire model will move to the CPU after use and applies float8 quantization to the transformer model, saving more memory.
-- sequential_cpu_offload means each layer of the model moves to CPU after use, which is slower but saves a lot of memory.
-
-qfloat8 may reduce model performance but saves more memory. If memory is sufficient, it's recommended to use model_cpu_offload.
-
-#### b. Via ComfyUI
-For more details, see the [ComfyUI README](https://github.com/aigc-apps/EasyAnimate/blob/main/comfyui/README.md).
-
-#### c. Run Python Files
-- Step 1: Download the corresponding [weights](#model-zoo) and place them in the models folder.
-- Step 2: Use different files for predictions based on the weights and prediction goals.
-  - Text-to-Video:
-    - Modify the prompt, neg_prompt, guidance_scale, and seed in the predict_t2v.py file.
-    - Then run the predict_t2v.py file and wait for the results, which are stored in the samples/easyanimate-videos folder.
-  - Image-to-Video:
-    - Modify validation_image_start, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_i2v.py file.
-    - validation_image_start is the starting image, and validation_image_end is the ending image of the video.
-    - Then run the predict_i2v.py file and wait for the results, which are stored in the samples/easyanimate-videos_i2v folder.
-  - Video-to-Video:
-    - Modify validation_video, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v.py file.
-    - validation_video is the reference video for video-to-video. You can run a demo with the following video: [Demo Video](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/play_guitar.mp4)
-    - Then run the predict_v2v.py file and wait for the results, which are stored in samples/easyanimate-videos_v2v folder.
-  - Generic Control Video (Canny, Pose, Depth, etc.):
-    - Modify control_video, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v_control.py file.
-    - control_video is the control video for video generation, extracted using Canny, Pose, Depth, etc. You can run a demo with the following video: [Demo Video](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1.1/pose.mp4)
-    - Then run the predict_v2v_control.py file and wait for the results, which are stored in samples/easyanimate-videos_v2v_control folder.
-  - Trajectory Control Video:
-    - Modify control_video, ref_image, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v_control.py file.
-    - control_video is the control video, and ref_image is the reference first frame image. You can run a demo with the following image and video: [Demo Image](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v5.1/dog.png), [Demo Video](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v5.1/trajectory_demo.mp4)
-    - Then run the predict_v2v_control.py file and wait for the results, which are stored in samples/easyanimate-videos_v2v_control folder.
-    - Interaction via ComfyUI is recommended.
-  - Camera Control Video:
-    - Modify control_video, ref_image, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the predict_v2v_control.py file.
-    - control_camera_txt is the control file for camera control video, and ref_image is the reference first frame image. You can run a demo with the following image and control file: [Demo Image](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/cogvideox_fun/asset/v1/firework.png), [Demo File (from CameraCtrl)](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v5.1/0a3b5fb184936a83.txt)
-    - Then run the predict_v2v_control.py file and wait for the results, which are stored in samples/easyanimate-videos_v2v_control folder.
-    - Interaction via ComfyUI is recommended.
-- Step 3: To combine with other backbones and Lora trained by yourself, modify predict_t2v.py and lora_path accordingly in the predict_t2v.py file.
-
-#### d. Via WebUI Interface
-
-WebUI supports text-to-video, image-to-video, video-to-video, and control-based video generation (such as Canny, Pose, Depth, etc.).
-
-- Step 1: Download the corresponding [weights](#model-zoo) and place them in the models folder.
-- Step 2: Run the app.py file to enter the Gradio page.
-- Step 3: Choose the generation model from the page, fill in prompt, neg_prompt, guidance_scale, seed, etc., click generate, and wait for the results, which are stored in the sample folder.
-
-# Quick Start
-### 1. Cloud usage: AliyunDSW/Docker
-#### a. From AliyunDSW
-DSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.
-
-Aliyun provide free GPU time in [Freetier](https://free.aliyun.com/?product=9602825&crowd=enterprise&spm=5176.28055625.J_5831864660.1.e939154aRgha4e&scm=20140722.M_9974135.P_110.MO_1806-ID_9974135-MID_9974135-CID_30683-ST_8512-V_1), get it and use in Aliyun PAI-DSW to start EasyAnimate within 5min!
-
-[![DSW Notebook](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/dsw.png)](https://gallery.pai-ml.com/#/preview/deepLearning/cv/easyanimate_v5)
-
-#### b. From ComfyUI
-Our ComfyUI is as follows, please refer to [ComfyUI README](https://github.com/aigc-apps/EasyAnimate/blob/main/comfyui/README.md) for details.
-![workflow graph](https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/v3/comfyui_i2v.jpg)
-
-#### c. From docker
-If you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.
-
-Then execute the following commands in this way:
+#### a、Text to video
+```python
+import torch
+import numpy as np
+from diffusers import EasyAnimatePipeline
+from diffusers.utils import export_to_video
+
+# Models: "alibaba-pai/EasyAnimateV5.1-7b-zh-diffusers" or "alibaba-pai/EasyAnimateV5.1-12b-zh-diffusers"
+pipe = EasyAnimatePipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = (
+    "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. "
+    "The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other "
+    "pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, "
+    "casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. "
+    "The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical "
+    "atmosphere of this unique musical performance."
+)
+negative_prompt     = "bad detailed"
+height              = 512
+width               = 512
+guidance_scale      = 6
+num_inference_steps = 50
+num_frames          = 49
+seed                = 43
+generator           = torch.Generator(device="cuda").manual_seed(seed)
+
+video = pipe(
+    prompt=prompt, 
+    negative_prompt=negative_prompt,
+    guidance_scale=guidance_scale, 
+    num_inference_steps=num_inference_steps,
+    num_frames=num_frames,
+    height=height,
+    width=width,
+    generator=generator,
+).frames[0]
+export_to_video(video, "output.mp4", fps=8)
 ```
-# pull image
-docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate
-
-# enter image
-docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate
-
-# clone code
-git clone https://github.com/aigc-apps/EasyAnimate.git
-
-# enter EasyAnimate's dir
-cd EasyAnimate
 
-# download weights
-mkdir models/Diffusion_Transformer
-mkdir models/Motion_Module
-mkdir models/Personalized_Model
-
-# Please use the hugginface link or modelscope link to download the EasyAnimateV5.1 model.
-# https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP
-# https://modelscope.cn/models/PAI/EasyAnimateV5.1-12b-zh-InP
+#### b、Image to video
+```python
+import torch
+from diffusers import EasyAnimateInpaintPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.utils import export_to_video, load_image, load_video
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers"
+pipe = EasyAnimateInpaintPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+
+validation_image_start = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
+validation_image_end = None
+sample_size = (448, 576)
+num_frames = 49
+
+input_video, input_video_mask = get_image_to_video_latent([validation_image_start], validation_image_end, num_frames, sample_size)
+
+video = pipe(
+    prompt, 
+    negative_prompt=negative_prompt,
+    num_frames=num_frames, 
+    height=sample_size[0], 
+    width=sample_size[1], 
+    video=input_video, 
+    mask_video=input_video_mask
+)
+export_to_video(video.frames[0], "output.mp4", fps=8)
+```
+#### c、Video to video
+```python
+import torch
+from diffusers import EasyAnimateInpaintPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.utils import export_to_video, load_image, load_video
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers"
+pipe = EasyAnimateInpaintPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = "一只穿着小外套的猫咪正安静地坐在花园的秋千上弹吉他。它的小外套精致而合身，增添了几分俏皮与可爱。晚霞的余光洒在它柔软的毛皮上，给它的毛发镀上了一层温暖的金色光辉。和煦的微风轻轻拂过，带来阵阵花香和草木的气息，令人心旷神怡。周围斑驳的光影随着音乐的旋律轻轻摇曳，仿佛整个花园都在为这只小猫咪的演奏伴舞。阳光透过树叶间的缝隙，投下一片片光影交错的图案，与悠扬的吉他声交织在一起，营造出一种梦幻而宁静的氛围。猫咪专注而投入地弹奏着，每一个音符都似乎充满了魔力，让这个傍晚变得更加美好。"
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+sample_size = (384, 672)
+num_frames = 49
+input_video = load_video("https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-InP/resolve/main/asset/1.mp4")
+input_video, input_video_mask, _ = get_video_to_video_latent(input_video, num_frames=num_frames, validation_video_mask=None, sample_size=sample_size)
+video = pipe(
+    prompt, 
+    num_frames=num_frames, 
+    negative_prompt=negative_prompt,
+    height=sample_size[0], 
+    width=sample_size[1], 
+    video=input_video, 
+    mask_video=input_video_mask,
+    strength=0.70
+)
+
+export_to_video(video.frames[0], "output.mp4", fps=8)
 ```
 
-### 2. Local install: Environment Check/Downloading/Installation
-#### a. Environment Check
-We have verified EasyAnimate execution on the following environment:
-
-The detailed of Windows:
-- OS: Windows 10
-- python: python3.10 & python3.11
-- pytorch: torch2.2.0
-- CUDA: 11.8 & 12.1
-- CUDNN: 8+
-- GPU： Nvidia-3060 12G
-
-The detailed of Linux:
-- OS: Ubuntu 20.04, CentOS
-- python: python3.10 & python3.11
-- pytorch: torch2.2.0
-- CUDA: 11.8 & 12.1
-- CUDNN: 8+
-- GPU：Nvidia-V100 16G & Nvidia-A10 24G & Nvidia-A100 40G & Nvidia-A100 80G
+#### d、Control to video
+```python
+import numpy as np
+import torch
+from diffusers import EasyAnimateControlPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.utils import export_to_video, load_video
+from PIL import Image
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-Control-diffusers"
+pipe = EasyAnimateControlPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+control_video = load_video(
+    "https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control/resolve/main/asset/pose.mp4"
+)
+prompt = (
+    "In this sunlit outdoor garden, a beautiful woman is dressed in a knee-length, sleeveless white dress. "
+    "The hem of her dress gently sways with her graceful dance, much like a butterfly fluttering in the breeze. "
+    "Sunlight filters through the leaves, casting dappled shadows that highlight her soft features and clear eyes, "
+    "making her appear exceptionally elegant. It seems as if every movement she makes speaks of youth and vitality. "
+    "As she twirls on the grass, her dress flutters, as if the entire garden is rejoicing in her dance. "
+    "The colorful flowers around her sway in the gentle breeze, with roses, chrysanthemums, and lilies each "
+    "releasing their fragrances, creating a relaxed and joyful atmosphere."
+)
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+sample_size = (672, 384)
+num_frames = 49
+generator = torch.Generator(device="cuda").manual_seed(43)
+input_video, _, _ = get_video_to_video_latent(np.array(control_video), num_frames, sample_size)
+
+video = pipe(prompt, num_frames=num_frames, negative_prompt=negative_prompt, height=sample_size[0], width=sample_size[1], control_video=input_video, generator=generator).frames[0]
+export_to_video(video, "output.mp4", fps=8)
+```
 
-We need about 60GB available on disk (for saving weights), please check!
+#### e、Camera Control to video
+Since the camera control model needs to process the camera files, please go to the [asset](https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera/blob/main/asset/) page to download the corresponding TXT files for camera movements.
+
+The relevant code is relatively complex and has been hidden. Please click to expand it.
+
+<details>
+  <summary>EasyAnimateV5.1:</summary>
+
+```python
+import numpy as np
+import torch
+from diffusers import EasyAnimateControlPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.utils import export_to_video, load_video, load_image
+from einops import rearrange
+from packaging import version as pver
+from PIL import Image
+
+
+class Camera(object):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    def __init__(self, entry):
+        fx, fy, cx, cy = entry[1:5]
+        self.fx = fx
+        self.fy = fy
+        self.cx = cx
+        self.cy = cy
+        w2c_mat = np.array(entry[7:]).reshape(3, 4)
+        w2c_mat_4x4 = np.eye(4)
+        w2c_mat_4x4[:3, :] = w2c_mat
+        self.w2c_mat = w2c_mat_4x4
+        self.c2w_mat = np.linalg.inv(w2c_mat_4x4)
+
+def custom_meshgrid(*args):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    # ref: https://pytorch.org/docs/stable/generated/torch.meshgrid.html?highlight=meshgrid#torch.meshgrid
+    if pver.parse(torch.__version__) < pver.parse('1.10'):
+        return torch.meshgrid(*args)
+    else:
+        return torch.meshgrid(*args, indexing='ij')
+
+def get_relative_pose(cam_params):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    abs_w2cs = [cam_param.w2c_mat for cam_param in cam_params]
+    abs_c2ws = [cam_param.c2w_mat for cam_param in cam_params]
+    cam_to_origin = 0
+    target_cam_c2w = np.array([
+        [1, 0, 0, 0],
+        [0, 1, 0, -cam_to_origin],
+        [0, 0, 1, 0],
+        [0, 0, 0, 1]
+    ])
+    abs2rel = target_cam_c2w @ abs_w2cs[0]
+    ret_poses = [target_cam_c2w, ] + [abs2rel @ abs_c2w for abs_c2w in abs_c2ws[1:]]
+    ret_poses = np.array(ret_poses, dtype=np.float32)
+    return ret_poses
+
+def ray_condition(K, c2w, H, W, device):
+    """Copied from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    # c2w: B, V, 4, 4
+    # K: B, V, 4
+
+    B = K.shape[0]
+
+    j, i = custom_meshgrid(
+        torch.linspace(0, H - 1, H, device=device, dtype=c2w.dtype),
+        torch.linspace(0, W - 1, W, device=device, dtype=c2w.dtype),
+    )
+    i = i.reshape([1, 1, H * W]).expand([B, 1, H * W]) + 0.5  # [B, HxW]
+    j = j.reshape([1, 1, H * W]).expand([B, 1, H * W]) + 0.5  # [B, HxW]
+
+    fx, fy, cx, cy = K.chunk(4, dim=-1)  # B,V, 1
+
+    zs = torch.ones_like(i)  # [B, HxW]
+    xs = (i - cx) / fx * zs
+    ys = (j - cy) / fy * zs
+    zs = zs.expand_as(ys)
+
+    directions = torch.stack((xs, ys, zs), dim=-1)  # B, V, HW, 3
+    directions = directions / directions.norm(dim=-1, keepdim=True)  # B, V, HW, 3
+
+    rays_d = directions @ c2w[..., :3, :3].transpose(-1, -2)  # B, V, 3, HW
+    rays_o = c2w[..., :3, 3]  # B, V, 3
+    rays_o = rays_o[:, :, None].expand_as(rays_d)  # B, V, 3, HW
+    # c2w @ dirctions
+    rays_dxo = torch.cross(rays_o, rays_d)
+    plucker = torch.cat([rays_dxo, rays_d], dim=-1)
+    plucker = plucker.reshape(B, c2w.shape[1], H, W, 6)  # B, V, H, W, 6
+    # plucker = plucker.permute(0, 1, 4, 2, 3)
+    return plucker
+
+def process_pose_file(pose_file_path, width=672, height=384, original_pose_width=1280, original_pose_height=720, device='cpu', return_poses=False):
+    """Modified from https://github.com/hehao13/CameraCtrl/blob/main/inference.py
+    """
+    with open(pose_file_path, 'r') as f:
+        poses = f.readlines()
+
+    poses = [pose.strip().split(' ') for pose in poses[1:]]
+    cam_params = [[float(x) for x in pose] for pose in poses]
+    if return_poses:
+        return cam_params
+    else:
+        cam_params = [Camera(cam_param) for cam_param in cam_params]
+
+        sample_wh_ratio = width / height
+        pose_wh_ratio = original_pose_width / original_pose_height  # Assuming placeholder ratios, change as needed
+
+        if pose_wh_ratio > sample_wh_ratio:
+            resized_ori_w = height * pose_wh_ratio
+            for cam_param in cam_params:
+                cam_param.fx = resized_ori_w * cam_param.fx / width
+        else:
+            resized_ori_h = width / pose_wh_ratio
+            for cam_param in cam_params:
+                cam_param.fy = resized_ori_h * cam_param.fy / height
+
+        intrinsic = np.asarray([[cam_param.fx * width,
+                                cam_param.fy * height,
+                                cam_param.cx * width,
+                                cam_param.cy * height]
+                                for cam_param in cam_params], dtype=np.float32)
+
+        K = torch.as_tensor(intrinsic)[None]  # [1, 1, 4]
+        c2ws = get_relative_pose(cam_params)  # Assuming this function is defined elsewhere
+        c2ws = torch.as_tensor(c2ws)[None]  # [1, n_frame, 4, 4]
+        plucker_embedding = ray_condition(K, c2ws, height, width, device=device)[0].permute(0, 3, 1, 2).contiguous()  # V, 6, H, W
+        plucker_embedding = plucker_embedding[None]
+        plucker_embedding = rearrange(plucker_embedding, "b f c h w -> b f h w c")[0]
+        return plucker_embedding
+    
+def get_image_latent(ref_image=None, sample_size=None):
+    if ref_image is not None:
+        if isinstance(ref_image, str):
+            ref_image = Image.open(ref_image).convert("RGB")
+            ref_image = ref_image.resize((sample_size[1], sample_size[0]))
+            ref_image = torch.from_numpy(np.array(ref_image))
+            ref_image = ref_image.unsqueeze(0).permute([3, 0, 1, 2]).unsqueeze(0) / 255
+        else:
+            ref_image = torch.from_numpy(np.array(ref_image))
+            ref_image = ref_image.unsqueeze(0).permute([3, 0, 1, 2]).unsqueeze(0) / 255
+
+    return ref_image
+
+# Models: "alibaba-pai/EasyAnimateV5.1-7b-zh-Control-Camera-diffusers" or "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera-diffusers"
+pipe = EasyAnimateControlPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+input_video, input_video_mask = None, None
+prompt = "Fireworks light up the evening sky over a sprawling cityscape with gothic-style buildings featuring pointed towers and clock faces. The city is lit by both artificial lights from the buildings and the colorful bursts of the fireworks. The scene is viewed from an elevated angle, showcasing a vibrant urban environment set against a backdrop of a dramatic, partially cloudy sky at dusk."
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+sample_size = (384, 672)
+num_frames = 49
+fps = 8
+ref_image = load_image("https://huggingface.co/alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera/resolve/main/asset/1.png")
+
+control_camera_video = process_pose_file("/The_Path_To/Pan_Left.txt", sample_size[1], sample_size[0])
+control_camera_video = control_camera_video[::int(24 // fps)][:num_frames].permute([3, 0, 1, 2]).unsqueeze(0)
+ref_image = get_image_latent(sample_size=sample_size, ref_image=ref_image)
+video = pipe(
+    prompt, 
+    negative_prompt=negative_prompt, 
+    num_frames=num_frames, 
+    height=sample_size[0], 
+    width=sample_size[1], 
+    control_camera_video=control_camera_video, 
+    ref_image=ref_image
+).frames[0]
+export_to_video(video, "output.mp4", fps=fps)
+```
+</details>
+
+#### f、float8 model
+Since the parameters of EasyAnimateV5.1 are very large, we need to consider memory-saving solutions to adapt to consumer-grade GPUs. We can convert the model to float8 to save GPU memory.
+
+For example, in the case of text-to-video generation, we first load the model in float8 to save memory, and then convert it to bfloat16 during inference.
+```python
+"""Modified from https://github.com/kijai/ComfyUI-MochiWrapper
+"""
+import torch
+import torch.nn as nn
+from diffusers import EasyAnimateInpaintPipeline
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_control import \
+    get_video_to_video_latent
+from diffusers.pipelines.easyanimate.pipeline_easyanimate_inpaint import \
+    get_image_to_video_latent
+from diffusers.utils import export_to_video, load_image, load_video
+
+def autocast_model_forward(cls, origin_dtype, *inputs, **kwargs):
+    weight_dtype = cls.weight.dtype
+    cls.to(origin_dtype)
+
+    # Convert all inputs to the original dtype
+    inputs = [input.to(origin_dtype) for input in inputs]
+    out = cls.original_forward(*inputs, **kwargs)
+
+    cls.to(weight_dtype)
+    return out
+
+def convert_weight_dtype_wrapper(module, origin_dtype):
+    for name, module in module.named_modules():
+        if name == "" or "embed_tokens" in name:
+            continue
+        original_forward = module.forward
+        if hasattr(module, "weight"):
+            setattr(module, "original_forward", original_forward)
+            setattr(
+                module,
+                "forward",
+                lambda *inputs, m=module, **kwargs: autocast_model_forward(m, origin_dtype, *inputs, **kwargs)
+            )
+
+# Models: "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers" or "alibaba-pai/EasyAnimateV5.1-7b-zh-InP-diffusers"
+pipe = EasyAnimateInpaintPipeline.from_pretrained(
+    "alibaba-pai/EasyAnimateV5.1-12b-zh-InP-diffusers", 
+    torch_dtype=torch.bfloat16
+)
+pipe.transformer = pipe.transformer.to(torch.float8_e4m3fn)
+from fp8_optimization import convert_weight_dtype_wrapper
+
+for _text_encoder in [pipe.text_encoder, pipe.text_encoder_2]:
+    if hasattr(_text_encoder, "visual"):
+        del _text_encoder.visual
+convert_weight_dtype_wrapper(pipe.transformer, torch.bfloat16)
+pipe.enable_model_cpu_offload()
+pipe.vae.enable_tiling()
+pipe.vae.enable_slicing()
+
+prompt = "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."
+negative_prompt = "Twisted body, limb deformities, text subtitles, comics, stillness, ugliness, errors, garbled text."
+validation_image_start = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
+validation_image_end = None
+sample_size = (448, 576)
+num_frames = 49
+input_video, input_video_mask = get_image_to_video_latent(
+    [validation_image_start], validation_image_end, num_frames, sample_size
+)
+
+video = pipe(
+    prompt, 
+    negative_prompt=negative_prompt,
+    num_frames=num_frames, 
+    height=sample_size[0], 
+    width=sample_size[1], 
+    video=input_video, 
+    mask_video=input_video_mask
+)
+export_to_video(video.frames[0], "output.mp4", fps=8)
+```
 
+# GPU Memory
 The video size for EasyAnimateV5.1-12B can be generated by different GPU Memory, including:
 | GPU memory | 384x672x25 | 384x672x49 | 576x1008x25 | 576x1008x49 | 768x1344x25 | 768x1344x49 |
 |------------|------------|------------|------------|------------|------------|------------|
@@ -340,9 +684,9 @@ The video size for EasyAnimateV5.1-7B can be generated by different GPU Memory,
 | 40GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 
 | 80GB | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 
 
-✅ indicates it can run under "model_cpu_offload", 🧡 represents it can run under "model_cpu_offload_and_qfloat8", ⭕️ indicates it can run under "sequential_cpu_offload", ❌ means it can't run. Please note that running with sequential_cpu_offload will be slower.
+✅ Indicates that it can run under "model_cpu_offload", 🧡 indicates that it can run under "model_cpu_offload" + float8, and ⭕️ indicates that it can run under "sequential_cpu_offload". Currently, due to the fact that qwen2vl does not support "sequential_cpu_offload", we are unable to use "sequential_cpu_offload" in diffusers. The specific fix time is unknown, but it can be used in the official EasyAnimate repository. ❌ Indicates that it cannot run. Please note that running with "sequential_cpu_offload" will be slower.
 
-Some GPUs that do not support torch.bfloat16, such as 2080ti and V100, require changing the weight_dtype in app.py and predict files to torch.float16 in order to run.
+Some GPUs that do not support torch.bfloat16, such as 2080ti and V100, require changing the torch.bfloat16 to torch.float16 in order to run.
 
 The generation time for EasyAnimateV5.1-12B using different GPUs over 25 steps is as follows:
 
@@ -351,23 +695,8 @@ The generation time for EasyAnimateV5.1-12B using different GPUs over 25 steps i
 | A10 24GB  | ~120s (4.8s/it)  | ~240s (9.6s/it)  | ~320s (12.7s/it) | ~750s (29.8s/it) | ❌               | ❌              |
 | A100 80GB | ~45s (1.75s/it)  | ~90s (3.7s/it)   | ~120s (4.7s/it)  | ~300s (11.4s/it) | ~265s (10.6s/it) | ~710s (28.3s/it) |
 
-#### b. Weights
-We'd better place the [weights](#model-zoo) along the specified path:
-
-EasyAnimateV5.1:
-```
-📦 models/
-├── 📂 Diffusion_Transformer/
-│   ├── 📂 EasyAnimateV5.1-12b-zh-InP/
-│   ├── 📂 EasyAnimateV5.1-12b-zh-Control/
-│   ├── 📂 EasyAnimateV5.1-12b-zh-Control-Camera/
-│   └── 📂 EasyAnimateV5.1-12b-zh/
-├── 📂 Personalized_Model/
-│   └── your trained trainformer model / your trained lora model (for UI load)
-```
-
 # Contact Us
-1. Use Dingding to search group 77450006752 or Scan to join
+1. Use Dingding to search group 77450006752 or Scan to join. 
 2. You need to scan the image to join the WeChat group or if it is expired, add this student as a friend first to invite you.
 
 <img src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/easyanimate/asset/group/dd.png" alt="ding group" width="30%"/>

transformer/config.json

@@ -11,6 +11,7 @@
   "dropout": 0.0,
   "enable_clip_in_inpaint": false,
   "enable_text_attention_mask": true,
+  "enable_mmdit_in_one_attn": true,
   "flip_sin_to_cos": true,
   "freq_shift": 0,
   "in_channels": 16,