mit-han-lab/svdquant-models

Quantization Library: DeepCompressor   Inference Engine: Nunchaku

[Paper]  [Code]  [Website]  [Blog]

SVDQuant seamlessly integrates with off-the-shelf LoRAs without requiring re-quantization. When applying LoRAs, it matches the image quality of the original 16-bit FLUX.1-dev.

Model Description

This reposity contains a converted LoRA collection for SVDQuant INT4 FLUX.1-dev. The LoRA style includes Realism, Ghibsky Illustration, Anime, Children Sketch, and Yarn Art.

Usage

Diffusers

Please follow the instructions in mit-han-lab/nunchaku to set up the environment. Then you can run the model with

import torch

from nunchaku.pipelines import flux as nunchaku_flux

pipeline = nunchaku_flux.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
    qmodel_path="mit-han-lab/svdq-int4-flux.1-dev",  # download from Huggingface
).to("cuda")
pipeline.transformer.nunchaku_update_params(mit-han-lab/svdquant-models/svdq-flux.1-dev-lora-anime.safetensors)
pipeline.transformer.nunchaku_set_lora_scale(1)
image = pipeline("a dog wearing a wizard hat", num_inference_steps=28, guidance_scale=3.5).images[0]
image.save("example.png")

Comfy UI

Work in progress.

Limitations

• The model is only runnable on NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See this issue for more details.
• You may observe some slight differences from the BF16 models in details.

Citation

If you find this model useful or relevant to your research, please cite

@article{
  li2024svdquant,
  title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models},
  author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song},
  journal={arXiv preprint arXiv:2411.05007},
  year={2024}
}