Project Zeus
The model weights belong to the Devil.
Download
Download the t5xxl_fp8_e4m3fn.safetensors from camenduru.
You should have:
project_zeus_sd3.beta_x.Q8.gguf
t5xxl_fp8_e4m3fn.safetensors
Setup
pip install diffusers numpy safetensors sentencepiece transformers
pip install "gguf @ git+https://github.com/ggerganov/llama.cpp.git@master#subdirectory=gguf-py"
Cooking pot
python convert_sd3_to_diffusers.py --checkpoint_path projectZeusSD3_betaX.safetensors --output project_zeus
python convert_to_gguf.py ./project_zeus/project_zeus.Q8.gguf ./project_zeus/transformer/diffusion_pytorch_model.safetensors
Inference
from diffusers import StableDiffusion3Pipeline, SD3Transformer2DModel
import gguf
from gguf.quants import dequantize
from safetensors.torch import load_file
from transformers import T5Config, T5EncoderModel
import torch
def load_transformer_from_gguf(filepath):
sd = {}
reader = gguf.GGUFReader(filepath)
for item in reader.tensors:
xs = dequantize(item.data, item.tensor_type)
tensor = torch.tensor(xs)
sd[item.name] = tensor.to(dtype=torch.float16)
config = SD3Transformer2DModel.load_config('./transformer')
transformer = SD3Transformer2DModel.from_config(config).to(dtype=torch.float16)
transformer.load_state_dict(sd)
transformer.eval()
return transformer
transformer = load_transformer_from_gguf('project_zeus_sd3.beta_x.Q8.gguf')
# Load T5 encoder in F8_E4M3 format.
# t5_config = T5Config.from_pretrained('text_encoder_3')
# t5 = T5EncoderModel(t5_config).to(dtype=torch.float16)
# t5_sd = load_file('t5xxl_fp8_e4m3fn.safetensors')
# t5.load_state_dict(t5_sd)
# t5.eval()
pipe = StableDiffusion3Pipeline.from_pretrained('stabilityai/stable-diffusion-3-medium-diffusers',
transformer=transformer,
# Either disable the T5 encoder.
text_encoder_3=None,
tokenizer_3=None,
# Or enable it.
# text_encoder_3=t5,
torch_dtype=torch.float16)
pipe.enable_model_cpu_offload()
image = pipe('cat playing piano', num_inference_steps=20).images[0]
image.save('cat.png')
Disclaimer
Use of this code and the copy of documentation requires citation and attribution to the author via a link to their Hugging Face profile in all resulting work.
- Downloads last month
- 111
Model tree for twodgirl/project-zeus-sd3-gguf
Base model
stabilityai/stable-diffusion-3-medium
Quantized
this model