Update README.md

README.md

@@ -5,12 +5,16 @@ license_link: LICENSE
 ---
 <!-- ## **HunyuanVideo** -->
 
+[中文阅读](./README_zh.md)
+
 <p align="center">
   <img src="https://raw.githubusercontent.com/Tencent/HunyuanVideo-I2V/refs/heads/main/assets/logo.png"  height=100>
 </p>
 
 # **HunyuanVideo-I2V** 🌅
 
+-----
+
 Following the great successful open-sourcing of our [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), we proudly present the [HunyuanVideo-I2V](https://github.com/Tencent/HunyuanVideo-I2V), a new image-to-video generation framework to accelerate open-source community exploration!
 
 This repo contains offical PyTorch model definitions, pre-trained weights and inference/sampling code. You can find more visualizations on our [project page](https://aivideo.hunyuan.tencent.com). Meanwhile, we have released the LoRA training code for customizable special effects, which can be used to create more interesting video effects.
@@ -20,15 +24,48 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
 
 
 ## 🔥🔥🔥 News!!
+* Mar 07, 2025: 🔥 We have fixed the bug in our open-source version that caused ID changes. Please try the new model weights of [HunyuanVideo-I2V](https://huggingface.co/tencent/HunyuanVideo-I2V) to ensure full visual consistency in the first frame and produce higher quality videos. 
 * Mar 06, 2025: 👋 We release the inference code and model weights of HunyuanVideo-I2V. [Download](https://github.com/Tencent/HunyuanVideo-I2V/blob/main/ckpts/README.md).
 
 
+### Frist Frame Consistency Demo
+|  Reference Image | Generated Video  |
+|:----------------:|:----------------:|
+|  <img src="https://github.com/user-attachments/assets/83e7a097-ffca-40db-9c72-be01d866aa7d" width="80%">   |       <video src="https://github.com/user-attachments/assets/f81d2c88-bb1a-43f8-b40f-1ccc20774563" width="100%"> </video>        | 
+｜ <img src="https://github.com/user-attachments/assets/c385a11f-60c7-4919-b0f1-bc5e715f673c" width="80%">         |       <video src="https://github.com/user-attachments/assets/0c29ede9-0481-4d40-9c67-a4b6267fdc2d" width="100%"> </video>        | 
+｜ <img src="https://github.com/user-attachments/assets/5763f5eb-0be5-4b36-866a-5199e31c5802" width="95%">         |       <video src="https://github.com/user-attachments/assets/a8da0a1b-ba7d-45a4-a901-5d213ceaf50e" width="100%"> </video>        |
+
+<!-- ### Customizable I2V LoRA Demo
+
+| I2V Lora Effect |  Reference Image | Generated Video  |
+|:---------------:|:--------------------------------:|:----------------:|
+|   Hair growth   |        <img src="./assets/demo/i2v_lora/imgs/hair_growth.png" width="40%">         |       <video src="https://github.com/user-attachments/assets/06b998ae-bbde-4c1f-96cb-a25a9197d5cb" width="100%"> </video>        |
+|     Embrace     |      <img src="./assets/demo/i2v_lora/imgs/embrace.png" width="40%">          |       <video src="https://github.com/user-attachments/assets/f8c99eb1-2a43-489a-ba02-6bd50a6dd260" width="100%" > </video>        |
+<!-- |   Hair growth   |        <img src="./assets/demo/i2v_lora/imgs/hair_growth.png" width="40%">         |       <video src="https://github.com/user-attachments/assets/06b998ae-bbde-4c1f-96cb-a25a9197d5cb" width="100%" poster="./assets/demo/i2v_lora/imgs/hair_growth.png"> </video>        |
+|     Embrace     |      <img src="./assets/demo/i2v_lora/imgs/embrace.png" width="40%">          |       <video src="https://github.com/user-attachments/assets/f8c99eb1-2a43-489a-ba02-6bd50a6dd260" width="100%" poster="./assets/demo/i2v_lora/imgs/hair_growth.png"> </video>        | -->
+
+<!-- ## 🧩 Community Contributions -->
+
+<!-- If you develop/use HunyuanVideo-I2V in your projects, welcome to let us know. -->
+
+<!-- - ComfyUI-Kijai (FP8 Inference, V2V and IP2V Generation): [ComfyUI-HunyuanVideoWrapper](https://github.com/kijai/ComfyUI-HunyuanVideoWrapper) by [Kijai](https://github.com/kijai) -->
+<!-- - ComfyUI-Native (Native Support): [ComfyUI-HunyuanVideo](https://comfyanonymous.github.io/ComfyUI_examples/hunyuan_video/) by [ComfyUI Official](https://github.com/comfyanonymous/ComfyUI) -->
+
+<!-- - FastVideo (Consistency Distilled Model and Sliding Tile Attention): [FastVideo](https://github.com/hao-ai-lab/FastVideo) and [Sliding Tile Attention](https://hao-ai-lab.github.io/blogs/sta/) by [Hao AI Lab](https://hao-ai-lab.github.io/)
+- HunyuanVideo-gguf (GGUF Version and Quantization): [HunyuanVideo-gguf](https://huggingface.co/city96/HunyuanVideo-gguf) by [city96](https://huggingface.co/city96)
+- Enhance-A-Video (Better Generated Video for Free): [Enhance-A-Video](https://github.com/NUS-HPC-AI-Lab/Enhance-A-Video) by [NUS-HPC-AI-Lab](https://ai.comp.nus.edu.sg/)
+- TeaCache (Cache-based Accelerate): [TeaCache](https://github.com/LiewFeng/TeaCache) by [Feng Liu](https://github.com/LiewFeng)
+- HunyuanVideoGP (GPU Poor version): [HunyuanVideoGP](https://github.com/deepbeepmeep/HunyuanVideoGP) by [DeepBeepMeep](https://github.com/deepbeepmeep)
+ --> 
+
+
+
 ## 📑 Open-source Plan
 - HunyuanVideo-I2V (Image-to-Video Model)
-  - [x] Lora training scripts
   - [x] Inference 
   - [x] Checkpoints
   - [x] ComfyUI
+  - [ ] Lora training scripts
   - [ ] Multi-gpus Sequence Parallel inference (Faster inference speed on more gpus)
   - [ ] Diffusers 
   - [ ] FP8 Quantified weight
@@ -44,20 +81,15 @@ This repo contains offical PyTorch model definitions, pre-trained weights and in
     - [Installation Guide for Linux](#installation-guide-for-linux)
   - [🧱 Download Pretrained Models](#-download-pretrained-models)
   - [🔑 Single-gpu Inference](#-single-gpu-inference)
+    - [Tips for Using Image-to-Video Models](#tips-for-using-image-to-video-models)
     - [Using Command Line](#using-command-line)
     - [More Configurations](#more-configurations)
-  - [🎉 Customizable I2V LoRA effects training](#-customizable-i2v-lora-effects-training)
-    - [Requirements](#requirements)
-    - [Environment](#environment)
-    - [Training data construction](#training-data-construction)
-    - [Training](#training)
-    - [Inference](#inference)
   - [🔗 BibTeX](#-bibtex)
   - [Acknowledgements](#acknowledgements)
 ---
 
 ## **HunyuanVideo-I2V Overall Architecture**
-Leveraging the advanced video generation capabilities of [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), we have extended its application to image-to-video generation tasks. To achieve this, we employ an image latent concatenation technique to effectively reconstruct and incorporate reference image information into the video generation process.
+Leveraging the advanced video generation capabilities of [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), we have extended its application to image-to-video generation tasks. To achieve this, we employ a token replace technique to effectively reconstruct and incorporate reference image information into the video generation process.
 
 Since we utilizes a pre-trained Multimodal Large Language Model (MLLM) with a Decoder-Only architecture as the text encoder, we can significantly enhance the model's ability to comprehend the semantic content of the input image and to seamlessly integrate information from both the image and its associated caption. Specifically, the input image is processed by the MLLM to generate semantic image tokens. These tokens are then concatenated with the video latent tokens, enabling comprehensive full-attention computation across the combined data.
 
@@ -135,7 +167,7 @@ docker run -itd --gpus all --init --net=host --uts=host --ipc=host --name hunyua
 
 ## 🧱 Download Pretrained Models
 
-The details of download pretrained models are shown [here](https://github.com/Tencent/HunyuanVideo-I2V/blob/main/ckpts/README.md).
+The details of download pretrained models are shown [here](ckpts/README.md).
 
 
 
@@ -143,6 +175,17 @@ The details of download pretrained models are shown [here](https://github.com/Te
 
 Similar to [HunyuanVideo](https://github.com/Tencent/HunyuanVideo), HunyuanVideo-I2V supports high-resolution video generation, with resolution up to 720P and video length up to 129 frames (5 seconds).
 
+### Tips for Using Image-to-Video Models
+- **Use Concise Prompts**: To effectively guide the model's generation, keep your prompts short and to the point.
+- **Include Key Elements**: A well-structured prompt should cover:
+  - **Main Subject**: Specify the primary focus of the video.
+  - **Action**: Describe the main movement or activity taking place.
+  - **Background (Optional)**: Set the scene for the video.
+  - **Camera Angle (Optional)**: Indicate the perspective or viewpoint.
+- **Avoid Overly Detailed Prompts**: Lengthy or highly detailed prompts can lead to unnecessary transitions in the video output.
+
+<!-- **For image-to-video models, we recommend using concise prompts to guide the model's generation process. A good prompt should include elements such as background, main subject, action, and camera angle. Overly long or excessively detailed prompts may introduce unnecessary transitions.** -->
+
 ### Using Command Line
 
 <!-- ### Run a Gradio Server
@@ -152,44 +195,68 @@ python3 gradio_server.py --flow-reverse
 # set SERVER_NAME and SERVER_PORT manually
 # SERVER_NAME=0.0.0.0 SERVER_PORT=8081 python3 gradio_server.py --flow-reverse
 ``` -->
+If you want to generate a more **stable** video, you can set `--i2v-stability` and `--flow-shift 7.0`. Execute the command as follows
 ```bash
 cd HunyuanVideo-I2V
 
 python3 sample_image2video.py \
+    --prompt "An Asian man with short hair in black tactical uniform and white clothes waves a firework stick." \
+    --i2v-image-path ./demo/imgs/0.jpg \
     --model HYVideo-T/2 \
-    --prompt "A man with short gray hair plays a red electric guitar." \
     --i2v-mode \
-    --i2v-image-path ./assets/demo/i2v/imgs/0.png \
     --i2v-resolution 720p \
+    --i2v-stability \
+    --infer-steps 50 \
     --video-length 129 \
+    --flow-reverse \
+    --flow-shift 7.0 \
+    --seed 0 \
+    --embedded-cfg-scale 6.0 \
+    --use-cpu-offload \
+    --save-path ./results
+```
+If you want to generate a more **high-dynamic** video, you can **unset** `--i2v-stability` and `--flow-shift 17.0`. Execute the command as follows
+```bash
+cd HunyuanVideo-I2V
+
+python3 sample_image2video.py \
+    --prompt "An Asian man with short hair in black tactical uniform and white clothes waves a firework stick." \
+    --i2v-image-path ./demo/imgs/0.jpg \
+    --model HYVideo-T/2 \
+    --i2v-mode \
+    --i2v-resolution 720p \
     --infer-steps 50 \
+    --video-length 129 \
     --flow-reverse \
     --flow-shift 17.0 \
     --seed 0 \
+    --embedded-cfg-scale 6.0 \
     --use-cpu-offload \
-    --save-path ./results 
+    --save-path ./results
 ```
 ### More Configurations
 
 We list some more useful configurations for easy usage:
 
-|        Argument        |            Default            |                          Description                          |
-|:----------------------:|:-----------------------------:|:------------------------------------------------------------:|
-|       `--prompt`       |             None              |           The text prompt for video generation.               |
-|       `--model`        |      HYVideo-T/2-cfgdistill   | Here we use HYVideo-T/2 for I2V, HYVideo-T/2-cfgdistill is used for T2V mode. |
-|     `--i2v-mode`       |            False              |                Whether to open i2v mode.                      |
-|  `--i2v-image-path`    | ./assets/demo/i2v/imgs/0.png  |        The reference image for video generation.              |
-|  `--i2v-resolution`    |            720p               |        The resolution for the generated video.                |
-|    `--video-length`    |             129               |         The length of the generated video.                    |
-|    `--infer-steps`     |              50               |         The number of steps for sampling.                     |
-|     `--flow-shift`     |             7.0               |     Shift factor for flow matching schedulers .               |
-|   `--flow-reverse`     |            False              | If reverse, learning/sampling from t=1 -> t=0.                |
-|        `--seed`        |             None              | The random seed for generating video, if None, we init a random seed. |
-|  `--use-cpu-offload`   |            False              | Use CPU offload for the model load to save more memory, necessary for high-res video generation. |
-|     `--save-path`      |         ./results             |         Path to save the generated video.                     |
+|        Argument        |           Default            |                                                                                        Description                                                                                        |
+|:----------------------:|:----------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+|       `--prompt`       |             None             |                                                                           The text prompt for video generation.                                                                           |
+|       `--model`        |    HYVideo-T/2-cfgdistill    |                                                       Here we use HYVideo-T/2 for I2V, HYVideo-T/2-cfgdistill is used for T2V mode.                                                       |
+|     `--i2v-mode`       |            False             |                                                                                 Whether to open i2v mode.                                                                                 |
+|  `--i2v-image-path`    | ./assets/demo/i2v/imgs/0.jpg |                                                                         The reference image for video generation.                                                                         |
+|  `--i2v-resolution`    |             720p             |                                                                          The resolution for the generated video.                                                                          |
+|  `--i2v-stability`    |            False             |                                                                       Whether to use stable mode for i2v inference.                                                                       |
+|    `--video-length`    |             129              |                                                                            The length of the generated video.                                                                             |
+|    `--infer-steps`     |              50              |                                                                             The number of steps for sampling.                                                                             |
+|     `--flow-shift`     |             7.0              | Shift factor for flow matching schedulers. We recommend 7 with `--i2v-stability` switch on for more stable video, 17 with `--i2v-stability` switch off for more dynamic video |
+|   `--flow-reverse`     |            False             |                                                                      If reverse, learning/sampling from t=1 -> t=0.                                                                       |
+|        `--seed`        |             None             |                                             The random seed for generating video, if None, we init a random seed.                                                                         |
+|  `--use-cpu-offload`   |            False             |                                             Use CPU offload for the model load to save more memory, necessary for high-res video generation.                                              |
+|     `--save-path`      |          ./results           |                                                                             Path to save the generated video.                                                                             |
+
 
 
-## 🎉 Customizable I2V LoRA effects training
+<!-- ## 🎉 Customizable I2V LoRA effects training
 
 ### Requirements
 
@@ -216,7 +283,7 @@ Prompt description: The trigger word is written directly in the video caption. I
 
 For example, AI hair growth effect (trigger): rapid_hair_growth, The hair of the characters in the video is growing rapidly. + original prompt
 
-After having the training video and prompt pair, refer to [here](https://github.com/Tencent/HunyuanVideo-I2V/blob/main/hyvideo/hyvae_extract/README.md) for training data construction.
+After having the training video and prompt pair, refer to [here](hyvideo/hyvae_extract/README.md) for training data construction.
 
 
 ### Training
@@ -259,7 +326,7 @@ We list some lora specific configurations for easy usage:
 |:-------------------:|:-------:|:----------------------------:|
 |    `--use-lora`     |  False  |  Whether to open lora mode.  |
 |   `--lora-scale`    |   1.0   | Fusion scale for lora model. |
-|   `--lora-path`     |   ""    |  Weight path for lora model. |
+|   `--lora-path`     |   ""    |  Weight path for lora model. | -->
 
 
 ## 🔗 BibTeX