mikaelh/flux-sanna-marin-lora-v0.3-fp8

Example image generated in ComfyUI using FLUX.1 [dev] model:

Example image generated in ComfyUI using FLUX.1 [schnell] model:

Third successful LoRA trained on FLUX.1 [dev] for 4950 steps. This LoRA was trained on all attention blocks, including the single attention blocks that deal with text inputs. See this commit for details: https://github.com/bghira/SimpleTuner/commit/a4f3385e9f7181cc1ea00c229aeb763c8a0a7a94

This LoRA was trained on Aug 8, 2024 but there have been many changes to SimpleTuner after that.

Trigger word: sanna marin

This version seems to have much bigger impact than previous v0.2 version when using the Dev model. The differences are rather small when using the Schnell model.

Trained using SimpleTuner: https://github.com/bghira/SimpleTuner

The base model and text encoders were fp8 quantized.

Example inference code using 🧨 diffusers (from inference.py):

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)

pipe.load_lora_weights("mikaelh/flux-sanna-marin-lora-v0.3-fp8", weight_name="pytorch_lora_weights.safetensors")

# Quantization is slow but necessary if VRAM is limited to 24 GB
if 1:
    from optimum.quanto import freeze, qfloat8, qint8, quantize

    weight_quant = qfloat8

    # Quantize transformer and text encoder similar to SimpleTuner
    quantize(pipe.transformer, weights=weight_quant)
    freeze(pipe.transformer)

    quantize(pipe.text_encoder, weights=weight_quant)
    freeze(pipe.text_encoder)

    quantize(pipe.text_encoder_2, weights=weight_quant)
    freeze(pipe.text_encoder_2)

pipe.enable_model_cpu_offload()

prompt = "closeup of sanna marin"
out = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    height=1024,
    width=1024,
    num_inference_steps=20,
).images[0]
out.save("image.png")

This LoRA is a derivative of the FLUX.1 [dev] model and therefore falls falls under the FLUX.1 [dev] Non-Commercial License.