README.md

---
datasets:
- diffusers/tuxemon
base_model:
- tencent/HunyuanVideo
library_name: diffusers
widget:
  - text: Style of snomexut, A fluffy, brown-haired tuxemon with a curious expression, sporting an orange and black striped harness and unique patterned tail, explores its surroundings.
    output:
      url: video_0.mp4
  - text: Style of snomexut, A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.
    output:
      url: video_1.mp4
  - text: Style of snomexut, A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.
    output:
      url: video_2.mp4
  - text: Style of snomexut, A menacing blue tuxemon with sharp claws and fierce fangs roars against a cosmic backdrop, its body adorned with glowing orbs and sharp pink details.
    output:
      url: video_3.mp4
tags:
- text-to-video
- diffusers-training
- diffusers
- lora
- hunyuan_video
- template:sd-lora
---

# LoRA Finetune

<Gallery />

## Model description

This is a lora finetune of https://huggingface.co./hunyuanvideo-community/HunyuanVideo.

The model was trained using [`finetrainers`](https://github.com/a-r-r-o-w/finetrainers). Training progress over time (4000 steps, 6000 steps, 8000 steps, 10000 steps):

<video src="https://huggingface.co./a-r-r-o-w/HunyuanVideo-tuxemons/resolve/main/output.mp4" autoplay controls loop> Your browser does not support the video tag. </video>

## Download model

[Download LoRA](pytorch_lora_weights.safetensors) in the Files & Versions tab.

## Usage

Requires the [🧨 Diffusers library](https://github.com/huggingface/diffusers) installed.

Trigger phrase: `"Style of snomexut,"`

```python
import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "hunyuanvideo-community/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
pipe.to("cuda")

pipe.load_lora_weights("a-r-r-o-w/HunyuanVideo-tuxemons", adapter_name="hunyuanvideo-lora")
pipe.set_adapters("hunyuanvideo-lora", 1.2)

output = pipe(
    prompt="Style of snomexut, a cat-like Tuxemon creature walks in alien-world grass, and observes its surroundings.",
    height=768,
    width=768,
    num_frames=33,
    num_inference_steps=30,
    generator=torch.Generator().manual_seed(73),
).frames[0]
export_to_video(output, "output-tuxemon.mp4", fps=15)
```

<video src="https://huggingface.co./a-r-r-o-w/HunyuanVideo-tuxemons/resolve/main/output-tuxemon.mp4" autoplay controls loop> Your browser does not support the video tag. </video>

For more details, including weighting, merging and fusing LoRAs, check the [documentation](https://huggingface.co./docs/diffusers/main/en/using-diffusers/loading_adapters) on loading LoRAs in diffusers.

<details>
<summary> Training params </summary>

Dataset contained 250 images from `diffusers/tuxemon` and 150 samples generated from a Flux LoRA trained on the same dataset.

```
id_token: "Style of snomexut,"
target_modules: to_q to_k to_v to_out.0 add_q_proj add_k_proj add_v_proj to_add_out
lr: 1e-5
training_steps: 10000
video_resolution_buckets: 1x768x768 1x512x512
rank: 128
lora_alpha: 128
optimizer: adamw
weight_decay: 0.01
flow_weighting_scheme: logit_normal
flow_shift: 7.0
validation_prompts: $ID_TOKEN A fluffy, brown-haired tuxemon with a curious expression, sporting an orange and black striped harness and unique patterned tail, explores its surroundings.@@@1x768x768:::$ID_TOKEN A curious yellow Tuxemon bird with large, expressive eyes and a distinctive golden beak, set against a starry black background, looks gently to its left.@@@1x768x768:::$ID_TOKEN A menacing Tuxemon with the appearance of a green and white aquatic shark, featuring sharp teeth and red eyes, emerges from the dark abyss, ready to unleash its fury.@@@1x768x768:::$ID_TOKEN A cheerful light blue tuxemon with four tentacle-like arms and sharp ears smiles playfully against a dark background.@@@1x768x768:::$ID_TOKEN A cheerful white Tuxemon with large ears and a playful expression joyfully floats in a starry outer space backdrop.@@@1x768x768:::$ID_TOKEN A finely detailed, silver-toned pin shaped like a futuristic double-barreled gun, adorned with orange textured panels, popularly associated with the Tuxemon universe.@@@1x768x768:::$ID_TOKEN A menacing blue tuxemon with sharp claws and fierce fangs roars against a cosmic backdrop, its body adorned with glowing orbs and sharp pink details.@@@1x768x768:::$ID_TOKEN A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.@@@1x768x768:::$ID_TOKEN A fluffy, brown-haired tuxemon with a curious expression, sporting an orange and black striped harness and unique patterned tail, explores its surroundings.@@@49x768x768:::$ID_TOKEN A curious yellow Tuxemon bird with large, expressive eyes and a distinctive golden beak, set against a starry black background, looks gently to its left.@@@49x768x768:::$ID_TOKEN A menacing Tuxemon with the appearance of a green and white aquatic shark, featuring sharp teeth and red eyes, emerges from the dark abyss, ready to unleash its fury.@@@49x768x768:::$ID_TOKEN A cheerful light blue tuxemon with four tentacle-like arms and sharp ears smiles playfully against a dark background.@@@49x768x768:::$ID_TOKEN A finely detailed, silver-toned pin shaped like a futuristic double-barreled gun, adorned with orange textured panels, popularly associated with the Tuxemon universe.@@@49x768x768:::$ID_TOKEN A menacing blue tuxemon with sharp claws and fierce fangs roars against a cosmic backdrop, its body adorned with glowing orbs and sharp pink details.@@@49x768x768:::$ID_TOKEN A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.@@@49x768x768
```
</details>