lmarena-ai/llama-3.2-sft-vision-arena

Llama-3.2-SFT-Vision-Arena Model Card

Model Details

Llama-3.2-SFT-Vision-Arena is a chat assistant trained by fine-tuning Llama-3.2-11B-Vision on user-shared conversations collected from Chatbot Arena.

Developed by: LMArena
Model type: An auto-regressive vision language model based on the transformer architecture
License: Llama 3.2 Community License Agreement
Finetuned from model: Llama-3.2-11B-Vision

Model Sources

Repository: https://github.com/lm-sys/FastChat
Paper: https://arxiv.org/abs/2412.08687

Sample Inference Code

import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor

model_id = "lmarena-ai/llama-3.2-sft-vision-arena"

model = MllamaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co./datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "Write a haiku about this image: "}
    ]}
]
input_text = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(
    image,
    input_text,
    add_special_tokens=False,
    return_tensors="pt"
).to(model.device)

output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0]))

Uses

The primary use of Llama-3.2-SFT-Vision-Arena is research on vision language models and chatbots. The primary intended users of the model are researchers and hobbyists in natural language processing, machine learning, and artificial intelligence.

BibTex

@misc{chou2024visionarena,
      title={VisionArena: 230K Real World User-VLM Conversations with Preference Labels}, 
      author={Christopher Chou and Lisa Dunlap and Koki Mashita and Krishna Mandal and Trevor Darrell and Ion Stoica and Joseph E. Gonzalez and Wei-Lin Chiang},
      year={2024},
      eprint={2412.08687},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2412.08687}, 
}