DGSpitzer/Guan-Yu-Diffusion

Guan Yu Diffusion

There is few model that focus on elements of Asian culture and heritage, so I manage to make my own one! Hope you like it! XD

Guan Yu Diffusion is an AI model that generates traditional oriental hero Guan Yu, from Three Kindoms Era in China during 220 to 280 AD!

As an ancient hero "Saint of War" and the Door God that ward off evil spirits, Guan Yu plays a big role in Asian culture, by chance you can find his statues and mini temples in a lot of Asian restaurants and markets all over the world.

It also comes with an additional support prompt to create Wu Xia - traditional Chinese martial art characters from movie and TV shows.

Based of Stable Diffusion V1.5, training in Fast Dreambooth

For research purposes only! This is an experiment I did to generate consistent character with clothing using AI - I used a series of photo of an 1/6 Guan Yu action figure model from Inflames Toy，with still photos from original Three Kingdom TV Show in 1994 as the dataset, therefore the model probably gonna overfit quite a lot by using only the default prompt keyword! (but you can then adding more prompts to reduce this issue)

by DGSpitzer

分享一个最近自己做的小实验，通过finetune实验训练了一个具有较高服装一致性的关羽AI生成模型Guan Yu Diffusion

原版的Stable Diffusion由于数据集的原因，对中国文化相关的人物生成效果比较薄弱，所以我试着训练了关羽人物的模型进行补充！

基于Diffusion模型的特性，于是还可以生成出各种经典的关羽形象，也可以与现代其他的英雄角色进行融合，实现时空穿越、东汉末年的自拍等等很有意思的效果。

为了能够保持AI生成人物服装统一，这次我采用的方法是通过导入同一个手办不同角度的照片，可以保证较高的服装连贯性。

训练用的数据集大部分来自INFLAMES TOYS 1/6神勇关云长兵人模型。因此使用初始词条生成的角色图片会非常接近手办本身（该模型请勿商用，仅作学习研究目的使用），可以通过加入其他的prompt词条，后置主词条的方式来弱化该模型过拟合的问题。

🧨 Diffusers

This repo contains both .ckpt and Diffuser model files. It's compatible to be used as any Stable Diffusion model, using standard Stable Diffusion Pipelines.

You can convert this model to ONNX, MPS and/or FLAX/JAX.

#!pip install diffusers transformers scipy torch
from diffusers import StableDiffusionPipeline
import torch

model_id = "DGSpitzer/Guan-Yu-Diffusion"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "Portrait of guanyu walking in ancient battlefield, close up shot"
image = pipe(prompt).images[0]

image.save("./guanyu.png")

Online Demo

You can try the Online Web UI demo build with Gradio, or use Colab Notebook at here:

My Online Space Demo

Buy me a coffee if you like this project ;P ♥

👇Model👇

AI Model Weights available at huggingface: https://huggingface.co./DGSpitzer/Guan-Yu-Diffusion/

Usage

After model loaded, use keyword guanyu in your prompt.

For sampler, use Euler A for the best result (DDIM kinda works too), CFG Scale 7, steps 20 should be fine

Example 1:

Portrait of guanyu walking in ancient battlefield, close up shot

You can merge the Guan Yu's outfit with other famous characters:

Example 2:

batman, portrait of fancy superhero guanyu, golden spiderman, mech, robot, high tech, shining core, intricate details, 4k

Example 3 (Additional keywork Wuxia):

There is another hidden keyword you can use in this model wuxia, to generate traditional Chinese martial art character:

a gorgeous wuxia girl standing in the palace

---

NOTE: usage of this model implies accpetance of stable diffusion's CreativeML Open RAIL-M license

---