library_name: diffusers
base_model:
- stabilityai/stable-diffusion-2-1-base
🍰 Hybrid-sd-small-vae for Stable Diffusion
Hybrid-sd-small-vae is a pruned-finetuned version VAE which uses the same "latent API" as the base model SD-VAE. It has smaller size, faster inference speed, as well as well-performed image generation in image saturation and image clarity compared to SD1.5. Specifically，we decreses parameters from original 83.65M to 62.01M, inferece time from 186.58ms to 135.58ms, roughly save up to 43.7% memory usage (12987MiB -> 9087MiB) without lossing T2I generation quality. The model is useful for real-time previewing of the SD1.x generation process, and you are very welcome to try it !!!!!!
Index Table
| Model | Params (M) | Decoder inference time (ms) | Decoder GPU Memory Usage (MiB) |
|---|---|---|---|
| SD1.5 | 83.65 | 186.58 | 12987 |
| Hybrid-sd-small-vae | 62.014 ↓ | 135.58 ↓ | 9087 ↓ |
T2I Comparison using one A100 GPU, The image order from left to right : SD-VAE -> Hybrid-sd-small-vae
This repo contains .safetensors versions of the Hybrid-sd-small-vae weights.
For SDXL, use Hybrid-sd-small-vae-xl instead (the SD and SDXL VAEs are incompatible).
Using in 🧨 diffusers
Firstly download our repository to load the AutoencoderKL
git clone https://github.com/bytedance/Hybrid-SD/tree/main
from bytenn_autoencoder_kl import AutoencoderKL
import torch
from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-base", torch_dtype=torch.float16
)
vae = AutoencoderKL.from_pretrained('cqyan/hybrid-sd-small-vae', torch_dtype=torch.float16)
pipe.vae = vae
pipe = pipe.to("cuda")
prompt = "A warm and loving family portrait, highly detailed, hyper-realistic, 8k resolution, photorealistic, soft and natural lighting"
image = pipe(prompt, num_inference_steps=25).images[0]
image.save("family.png")