LivePortrait
ONNX

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

Jianzhu Guo 1†  Dingyun Zhang 1,2  Xiaoqiang Liu 1  Zhizhou Zhong 1,3  Yuan Zhang 1 
Pengfei Wan 1  Di Zhang 1 
1 Kuaishou Technology  2 University of Science and Technology of China  3 Fudan University 
† Corresponding author

showcase πŸ”₯ For more results, visit our homepage πŸ”₯

πŸ”₯ Updates

  • 2024/08/02: 😸 We released a version of the Animals model, along with several other updates and improvements. Check out the details here!
  • 2024/07/25: πŸ“¦ Windows users can now download the package from HuggingFace or BaiduYun. Simply unzip and double-click run_windows.bat to enjoy!
  • 2024/07/24: 🎨 We support pose editing for source portraits in the Gradio interface. We’ve also lowered the default detection threshold to increase recall. Have fun!
  • 2024/07/19: ✨ We support 🎞️ portrait video editing (aka v2v)! More to see here.
  • 2024/07/17: 🍎 We support macOS with Apple Silicon, modified from jeethu's PR #143.
  • 2024/07/10: πŸ’ͺ We support audio and video concatenating, driving video auto-cropping, and template making to protect privacy. More to see here.
  • 2024/07/09: πŸ€— We released the HuggingFace Space, thanks to the HF team and Gradio!
  • 2024/07/04: 😊 We released the initial version of the inference code and models. Continuous updates, stay tuned!
  • 2024/07/04: πŸ”₯ We released the homepage and technical report on arXiv.

Introduction πŸ“–

This repo, named LivePortrait, contains the official PyTorch implementation of our paper LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) πŸ’–.

Getting Started 🏁

1. Clone the code and prepare the environment

git clone https://github.com/KwaiVGI/LivePortrait
cd LivePortrait

# create env using conda
conda create -n LivePortrait python==3.9
conda activate LivePortrait

# install dependencies with pip
# for Linux and Windows users
pip install -r requirements.txt
# for macOS with Apple Silicon users
pip install -r requirements_macOS.txt

Note: make sure your system has FFmpeg installed, including both ffmpeg and ffprobe!

2. Download pretrained weights

The easiest way to download the pretrained weights is from HuggingFace:

# first, ensure git-lfs is installed, see: https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage
git lfs install
# clone and move the weights
git clone https://huggingface.co./KwaiVGI/LivePortrait temp_pretrained_weights
mv temp_pretrained_weights/* pretrained_weights/
rm -rf temp_pretrained_weights

Alternatively, you can download all pretrained weights from Google Drive or Baidu Yun. Unzip and place them in ./pretrained_weights.

Ensuring the directory structure is as follows, or contains:

pretrained_weights
β”œβ”€β”€ insightface
β”‚   └── models
β”‚       └── buffalo_l
β”‚           β”œβ”€β”€ 2d106det.onnx
β”‚           └── det_10g.onnx
└── liveportrait
    β”œβ”€β”€ base_models
    β”‚   β”œβ”€β”€ appearance_feature_extractor.pth
    β”‚   β”œβ”€β”€ motion_extractor.pth
    β”‚   β”œβ”€β”€ spade_generator.pth
    β”‚   └── warping_module.pth
    β”œβ”€β”€ landmark.onnx
    └── retargeting_models
        └── stitching_retargeting_module.pth

3. Inference πŸš€

Fast hands-on

# For Linux and Windows
python inference.py

# For macOS with Apple Silicon, Intel not supported, this maybe 20x slower than RTX 4090
PYTORCH_ENABLE_MPS_FALLBACK=1 python inference.py

If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4. This file includes the following results: driving video, input image or video, and generated result.

image

Or, you can change the input by specifying the -s and -d arguments:

# source input is an image
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4

# source input is a video ✨
python inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving/d0.mp4

# more options to see
python inference.py -h

Driving video auto-cropping πŸ“’πŸ“’πŸ“’

To use your own driving video, we recommend: ⬇️

  • Crop it to a 1:1 aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-cropping by --flag_crop_driving_video.
  • Focus on the head area, similar to the example videos.
  • Minimize shoulder movement.
  • Make sure the first frame of driving video is a frontal face with neutral expression.

Below is a auto-cropping case by --flag_crop_driving_video:

python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d13.mp4 --flag_crop_driving_video

If you find the results of auto-cropping is not well, you can modify the --scale_crop_driving_video, --vy_ratio_crop_driving_video options to adjust the scale and offset, or do it manually.

Motion template making

You can also use the auto-generated motion template files ending with .pkl to speed up inference, and protect privacy, such as:

python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d5.pkl # portrait animation
python inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving/d5.pkl # portrait video editing

4. Gradio interface πŸ€—

We also provide a Gradio interface for a better experience, just run by:

# For Linux and Windows users (and macOS with Intel??)
python app.py

# For macOS with Apple Silicon users, Intel not supported, this maybe 20x slower than RTX 4090
PYTORCH_ENABLE_MPS_FALLBACK=1 python app.py

You can specify the --server_port, --share, --server_name arguments to satisfy your needs!

πŸš€ We also provide an acceleration option --flag_do_torch_compile. The first-time inference triggers an optimization process (about one minute), making subsequent inferences 20-30% faster. Performance gains may vary with different CUDA versions.

# enable torch.compile for faster inference
python app.py --flag_do_torch_compile

Note: This method is not supported on Windows and macOS.

Or, try it out effortlessly on HuggingFace πŸ€—

5. Inference speed evaluation πŸš€πŸš€πŸš€

We have also provided a script to evaluate the inference speed of each module:

# For NVIDIA GPU
python speed.py

Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with torch.compile:

Model Parameters(M) Model Size(MB) Inference(ms)
Appearance Feature Extractor 0.84 3.3 0.82
Motion Extractor 28.12 108 0.84
Spade Generator 55.37 212 7.59
Warping Module 45.53 174 5.21
Stitching and Retargeting Modules 0.23 2.3 0.31

Note: The values for the Stitching and Retargeting Modules represent the combined parameter counts and total inference time of three sequential MLP networks.

Community Resources πŸ€—

Discover the invaluable resources contributed by our community to enhance your LivePortrait experience:

And many more amazing contributions from our community!

Acknowledgements πŸ’

We would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace repositories, for their open research and contributions.

Citation πŸ’–

If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:

@article{guo2024liveportrait,
  title   = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},
  author  = {Guo, Jianzhu and Zhang, Dingyun and Liu, Xiaoqiang and Zhong, Zhizhou and Zhang, Yuan and Wan, Pengfei and Zhang, Di},
  journal = {arXiv preprint arXiv:2407.03168},
  year    = {2024}
}

Contact πŸ“§

Jianzhu Guo (郭建珠); [email protected]

Downloads last month
3,735
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Spaces using KwaiVGI/LivePortrait 8