README.md

---
license: creativeml-openrail-m
language:
- en
library_name: diffusers
pipeline_tag: text-to-video
tags:
- AIGC
- text2video
- image2video
- infinite-length
- human
---


<font size=5>**MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Parallel Denoising**</font>
</br>
Zhiqiang Xia <sup>\*</sup>,
Zhaokang Chen<sup>\*</sup>,
Bin Wu<sup>†</sup>,
Chao Li,
Kwok-Wai Hung,
Chao Zhan,
Wenjiang Zhou
(<sup>*</sup>Equal Contribution, <sup>†</sup>Corresponding Author)
</br>
Lyra Lab, Tencent Music Entertainment

**[project](comming soon)**  **Technical report (comming soon)**


We have setup **the world simulator vision since March 2023, believing diffusion models can simulate the world**. `MuseV` was a milestone achieved around **July 2023**. Amazed by the progress of Sora, we decided to opensource `MuseV`, hopefully it will benefit the community. Next we will move on to the promising diffusion+transformer scheme.

We will soon release `MuseTalk`, a real-time high quality lip sync model, which can be applied with MuseV as a complete virtual human generation solution. Please stay tuned! 

# Intro
`MuseV` is a diffusion-based virtual human video generation framework, which 
1. supports **infinite length** generation using a novel **Parallel Denoising scheme**.
2. checkpoint available for virtual human video generation trained on human dataset.
3. supports Image2Video, Text2Image2Video, Video2Video.
4. compatible with the **Stable Diffusion ecosystem**, including `base_model`, `lora`, `controlnet`, etc. 
5. supports multi reference image technology, including `IPAdapter`, `ReferenceOnly`, `ReferenceNet`, `IPAdapterFaceID`.
6. training codes (comming very soon).


## Model
### overview of model structure
![model_structure](data/models/musev_structure.png)
### parallel denoising
![parallel_denoise](data/models/parallel_denoise.png)

## Cases
All frames are generated from text2video model, without any post process.

Bellow Case could be found in `configs/tasks/example.yaml`
### Text/Image2Video

#### Human
 <!-- 2 columns, one image, one video -->

<table class="center">
  <tr style="font-weight: bolder;text-align:center;">
        <td>image</td>
        <td>video </td>
        <td>prompt</td>
  </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/cTQX49v7GT7GA-NEHj5vK.jpeg width="400">
    </td>
    <td >
     <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/U4sHVv_fYVbFHveS7Sw7h.mp4"></video>
    </td>
    <td>(masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/SPSgJpptVM4Qm11nqD07C.jpeg width="400">
    </td>
    <td>
    <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/XMya1FCmRs6USzKp9qrAy.mp4"></video>
    </td>
    <td>
    (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/l6chBPhUKeOLbnXnX-ewG.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/pPnU6QXgWuWxw5SWZdl8N.mp4"></video>
    </td>   
    <td>
    (masterpiece, best quality, highres:1), peaceful beautiful sea scene
    </td>
  </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/xKJnKg7AQEJCTzt4DW4n3.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/KezbR3_yWus5JFJdMlaV7.mp4"></video>
    </td>
    <td>
      (masterpiece, best quality, highres:1), peaceful beautiful sea scene
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/VQMEeGc1wTuiATtQLJjer.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/_9ZQEebUlmSNtXMJKiGPu.mp4"></video>
    </td>
    <td>
      (masterpiece, best quality, highres:1), peaceful beautiful sea scene
    </td>
  </tr>
  <!-- guitar  -->
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/Fk_eec7vqq4NfAYVPNLI-.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/p0gWRwZTDOrbPf8mZOphG.mp4"></video>
    </td>
    <td>
       (masterpiece, best quality, highres:1), playing guitar
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/Zcc5xj1-lA_EPS7gvJu99.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/4mT4KL3q4FzyQQKJfgXVG.mp4"></video>
    </td>
    <td>
      (masterpiece, best quality, highres:1), playing guitar
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/iT5OsCpRNnntuS0TH1cG5.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/6RWBw73-oE4rJH808FzIK.mp4">
    </td>
    <td>
      (masterpiece, best quality, highres:1), playing guitar
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/Ym582ZF-MbYkRW1sAE5r3.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/lYqpeRcIiK7WEqRe4d8dZ.mp4"></video>
    </td>
    <td>
    (masterpiece, best quality, highres:1), playing guitar
    </td>
  </tr>
  <!-- famous people -->
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/JsAjbl4AeYz089kWHjjUJ.jpeg width="400">
    </td>
    <td>
      <video width="500"  controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/KCjF9HutBo7el15gm3YV3.mp4"></video>
    </td>
    <td>
    (masterpiece, best quality, highres:1),(1man, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/v-X3Wrkm14YwLGGloNlMK.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/P_Y5jUO1EJ6n3Z4qd1xh1.mp4"></video>
    </td>   
    <td>
    (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>
  <tr>
    <td>
        <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/cNe41NV1OfLF5AmMKD6mi.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/L6mA8uuckRJhzhAJHayJT.mp4"></video>      
    </td>
    <td>
  (masterpiece, best quality, highres:1),(1man, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/6iHIaa15eBgop7BsE0Nps.jpeg width="400">
    </td>
    <td>
      <video width="500"  controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/foPX3iRk2TzjRl_V52T21.mp4"></video>
    </td>
    <td>
  (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>
  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/R7rp7t4DPkws0dXRxi0bf.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/tGTSNe9i08pvTMNe6SOBg.mp4"></video>
    </td>
    <td>
      (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
    soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
    </td>
  </tr>
</table >

#### scene

<table class="center">
    <tr style="font-weight: bolder;text-align:center;">
        <td width="35%">image</td>
        <td width="50%">video</td>
        <td width="15%">prompt</td>
    </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/EIembLBwySZTBjFZStFr_.jpeg width="400">
    </td>
    <td>
        <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/jfHV6a186BzAu-Tz0ET1o.mp4"></video>
    </td>
    <td>
      (masterpiece, best quality, highres:1), peaceful beautiful waterfall, an
    endless waterfall
    </td>
  </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/u2_mzl5m-Z0nwSYFcTLxs.jpeg width="400">
    </td>
    <td>
        <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/eXrRVejCZs3QaA4-JK6Le.mp4"></video>
    </td>
    <td>(masterpiece, best quality, highres:1), peaceful beautiful river
    </td>
  </tr>

  <tr>
    <td>
      <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/NIHfIi7onyJ5ELetE2f_Z.jpeg width="400">
    </td>
    <td>
      <video width="500" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/eaZyhukcfaoY7dIGp2cs6.mp4"></video>
    </td>
    <td>(masterpiece, best quality, highres:1), peaceful beautiful sea scene
    </td>
  </tr>
</table >

### VideoMiddle2Video



# News
- [03/22/2024] release `MuseV` project and trained model `musev`, `muse_referencenet`.


# Quickstart
please refer to [MuseV](https://github.com/TMElyralab/MuseV)


# Acknowledgements

MuseV builds on `TuneAVideo`, `diffusers`. Thanks  for open-sourcing!

<!-- # Contribution 暂时不需要组织开源共建 -->

# Citation
```bib
@article{musev,
  title={MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Parallel Denoising},
  author={Xia, Zhiqiang and Chen, Zhaokang and Wu, Bin and Li, Chao and Hung, Kwok-Wai and Zhan, Chao and Zhou, Wenjiang},
  journal={arxiv},
  year={2024}
}
```
# Disclaimer/License
1. `code`: The code of MuseV is released under the MIT License. There is no limitation for both academic and commercial usage.
1. `model`: The trained model are available for non-commercial research purposes only.
1. `other opensource model`: Other open-source models used must comply with their license, such as `insightface`, `IP-Adapter`, `ft-mse-vae`, etc.
1. `AIGC`: This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.