jakebabbidge/sdxl-adapter-inpaint

A custom pipeline to add inpainting support to T2I-Adapters with the SDXL model.

To use T2I-Adapters with SDXL, ensure you have diffusers installed via the t2iadapterxl branch like so:

pip install git+https://github.com/huggingface/diffusers.git@t2iadapterxl

The following is an example on how to use this pipeline along with a sketch T2I-Adapter:

>>> import torch
>>> from diffusers import DiffusionPipeline, T2IAdapter
>>> from diffusers.utils import load_image
>>> from PIL import Image

>>> adapter = T2IAdapter.from_pretrained(
... 	"TencentARC/t2i-adapter-sketch-sdxl-1.0", torch_dtype=torch.float16, variant="fp16"
... ).to("cuda")

>>> pipe = DiffusionPipeline.from_pretrained(
... 	"stabilityai/stable-diffusion-xl-base-1.0",
... 	torch_dtype=torch.float16,
... 	variant="fp16",
... 	use_safetensors=True,
... 	custom_pipeline="jakebabbidge/sdxl-adapter-inpaint",
... 	adapter=adapter
... ).to("cuda")

>>> image = Image.open(image_path).convert("RGB")
>>> mask = Image.open(mask_path).convert("RGB")
>>> adapter_sketch = Image.open(adapter_sketch_path).convert("RGB")

>>> result_image = pipe(
... 	image=image,
... 	mask_image=mask,
... 	adapter_image=adapter_sketch,
... 	prompt="a photo of a dog in real world, high quality",
... 	negative_prompt="extra digit, fewer digits, cropped, worst quality, low quality",
...	num_inference_steps=50
... ).images[0]