Instella-VL✨
Collection
2 items
•
Updated
Instella-VL-1B ✨
Welcome to the official repository for Instella-VL-1B, AMD's first ever Vision-Language Model (VLM). This repository provides a detailed guide for training and inference with Instella-VL-1B. Developed from AMD's Instella-1B (previously known as AMD OLMo 1B SFT LLM), this model is fully open-source, with both model weights and training code available for AMD GPUs (MI300). Its compact size aims to make it accessible to a broad spectrum of researchers, developers, and enthusiasts, enabling them to build upon, modify, and integrate it into their own projects.
Main Results
We compare our model with models which only releases the model weights (with * in the below table) and also models which releases weights, data curvation and all training details.
| DeepSeek-VL-1.3B * | InternVL2-1B * | InternVL2.5-1B * | TinyLLaVA-2.4B | TinyLLaVA-1.5B | llava-onevision-1b | MiniCPM-V-2 | Instella-VL-1B | |
| GQA | -- | 55.06 | 56.66 | 61.58 | 60.28 | 57.95 | -- | 61.52 |
| SQA | 64.52 | 89.54 | 93.90 | 64.30 | 59.69 | 59.25 | 76.10 | 83.74 |
| POPE | 85.80 | 87.40 | 89.95 | 85.66 | 84.77 | 87.17 | 86.56 | 86.73 |
| MM-Bench | 64.34 | 61.70 | 68.40 | 58.16 | 51.28 | 44.60 | 70.44 | 69.17 |
| seedbench | 65.94 | 65.90 | 71.30 | 63.30 | 60.04 | 65.43 | 66.90 | 68.47 |
| MMMU | 28.67 | 32.40 | 35.60 | 32.11 | 29.89 | 30.90 | 38.55 | 29.30 |
| realworldqa | 50.20 | 51.90 | 58.30 | 52.42 | 46.67 | 51.63 | 55.03 | 58.82 |
| mmstar | 38.30 | 46.18 | 47.93 | 37.17 | 31.87 | 37.38 | 40.93 | 43.21 |
| Average | - | 61.26 | 65.26 | 56.84 | 53.06 | 54.29 | - | 62.62 |
| ocrbench | 41.40 | 74.40 | 74.20 | 28.90 | 34.40 | 43.00 | 60.00 | 67.90 |
| TextVQA | 57.54 | 69.60 | 72.96 | 47.05 | 49.54 | 49.54 | 74.23 | 71.23 |
| AI2D | 51.13 | 62.40 | 67.58 | 49.58 | 43.10 | 57.35 | 64.40 | 66.65 |
| ChartQA | 47.40 | 71.52 | 75.76 | 12.96 | 15.24 | 61.24 | 59.80 | 72.52 |
| DocVQA | 35.70 | 80.94 | 82.76 | 25.82 | 30.38 | 71.22 | 69.54 | 80.30 |
| InfoVQA | 20.52 | 46.30 | 53.62 | 21.35 | 24.46 | 41.18 | 38.24 | 46.40 |
| OCR Average | 42.28 | 67.53 | 71.15 | 30.94 | 32.85 | 53.92 | 61.04 | 67.50 |
Quick Start
Follow below packages list for setting up the inference environment.
pip==25.0 wheel==0.45.1 setuptools==75.8.0 torch==2.6.0 torchvision==0.21.0 transformers==4.49.0 einops==0.8.0
import torch
from transformers import AutoTokenizer, AutoProcessor, AutoConfig, AutoModelForCausalLM
from PIL import Image
import requests
from io import BytesIO
def load_image(image_file):
if image_file.startswith("http") or image_file.startswith("https"):
response = requests.get(image_file)
image = Image.open(BytesIO(response.content)).convert("RGB")
else:
image = Image.open(image_file).convert("RGB")
return image
config = AutoConfig.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("amd/Instella-VL-1B", config=config, trust_remote_code=True)
processor = AutoProcessor.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True).to('cuda') # or 'cpu'
model.eval()
# For single image and text
query="Describe the image."
image=load_image("path/to/your_image") # can be a https:// url
out = processor.encode(query, image, model.get_vision_tower().image_processor, tokenizer, config)
inputs = {k: v.to(model.device) for k, v in out.items() if isinstance(v, torch.Tensor)}
with torch.inference_mode():
output_ids = model.generate(inputs["input_ids"], images=inputs['image_tensor'], image_sizes=out['image_sizes'], do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=out['stopping_criteria'], eos_token_id=out['eos_token_id'])
outputs = processor.decode(output_ids)
print("InstellaVL: ", outputs)
# For batch of images and text.
query=["Describe the image.", "What is the color of the dog?"]
image=[load_image("../assets/images/instellavl.png"), load_image("../assets/images/example2_dog.jpg")]
outs = processor.batch_encode(query, image, model.get_vision_tower().image_processor, tokenizer, config)
for idx, o in enumerate(outs):
ins = {k: v.to(model.device) for k, v in o.items() if isinstance(v, torch.Tensor)}
with torch.inference_mode():
output_ids = model.generate(ins["input_ids"],
images=ins['image_tensor'],
image_sizes=o['image_sizes'],
do_sample=True,
num_beams=1,
temperature=0.2,
max_new_tokens=1024,
use_cache=True,
stopping_criteria=o['stopping_criteria'],
eos_token_id=o['eos_token_id'])
outputs = processor.decode(output_ids)
print("Query: ", query[idx])
print("InstellaVL: ", outputs)
TL;DR: Loading from locally saved checkpoint
Note: Do pip install -e . --no-deps to register/include for InstellaVL repo as instellavl package into Python package list.
import torch
# Import essential modules
from instellavl.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX
from instellavl.conversation import conv_templates, SeparatorStyle
from instellavl.model.builder import load_pretrained_model
from instellavl.utils import disable_torch_init
from instellavl.mm_utils import process_images, tokenizer_image_token, get_model_name_from_path
from PIL import Image
import requests
from io import BytesIO
# Login into HF Hub
from huggingface_hub import login
login(token = "<Your HFtoken id>") # Enter your token
def load_image(image_file):
if image_file.startswith("http") or image_file.startswith("https"):
response = requests.get(image_file)
image = Image.open(BytesIO(response.content)).convert("RGB")
else:
image = Image.open(image_file).convert("RGB")
return image
#
# ========= CHANGE IMAGE and Query only HERE ============
image_file = '/path/to/Instella-VL-repo/assets/images/example2_dog.jpg' # Enter the test image path
query = 'Describe this image.'
# =======================================================
disable_torch_init()
conv_mode = 'instella'
# Model loading
model_path = '<path/to/model-checkpoint-saved-locally>' # Enter your model path, should contain instellavl substring in the name.
model_name = get_model_name_from_path(model_path)
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name, False, False)
model.eval()
model = model.to('cuda') # change to 'cpu' if not 'cuda'
# Image pre-processing
image = load_image(image_file)
image_tensor = process_images([image], image_processor, model.config)
image_tensor = image_processor.preprocess(image, return_tensors="pt")["pixel_values"].to(model.dtype)
# Text pre-processing - follow the below logic too when there is no Image:
# if images is not None and len(image_tensor) != 0 and DEFAULT_IMAGE_TOKEN not in text:
# question = DEFAULT_IMAGE_TOKEN + "\n" + text
# else:
# question = text
query = query.replace(DEFAULT_IMAGE_TOKEN, "").strip()
question = DEFAULT_IMAGE_TOKEN + "\n" + query
conv = conv_templates[conv_mode].copy()
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()
# Final arrangements required
input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0)
keywords = [conv.sep]
image_sizes = [image.size]
stopping_criteria = [KeywordsStoppingCriteria(keywords, tokenizer, input_ids)]
terminators = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("|||IP_ADDRESS|||")]
with torch.inference_mode():
output_ids = model.generate(input_ids.to(model.device), images=image_tensor.to(model.device), image_sizes=image_sizes, do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=stopping_criteria, eos_token_id=terminators)
outputs = tokenizer.decode(output_ids[0, input_ids.shape[1] :]).strip()
print("InstellaVL: ", outputs)
Model Architecture
| Parts | Parameter size | Number of layers | Number of heads | Hidden size | Patch Size |
|---|---|---|---|---|---|
| Vision Encoder | 300M | 24 | 16 | 1024 | 14 |
| MLP | 6.3M | 2 | - | 2048 | - |
| LM | 1.2B | 16 | 16 | 2048 | - |
We initialize the vision encoder from CLIP-ViT-L/14@336 and initialize LM from AMD OLMo 1B SFT
Training Stages
| Stages | MLP Warmup | Pretraining | Instruction Tuning |
|---|---|---|---|
| Tunable Parts | Adapter | Entire Model | Entire Model |
Hardware
Training was conducted with up to 4 nodes, totaling 32 GPUs. Each node comprises 8 AMD Instinct™ MI300X GPUs
MLP warmup: 1 node
Pretraining: 2 nodes
Finetune: 4 nodes
Datasets
MLP Warmup
Pretraining Stage
| Domain | Datasets | Num of Examples | Licenses |
|---|---|---|---|
| Image Captions | BLIP150K, COCO118K, CC3M-Recap, Pixmo_Cap | 3.52M | BSD 3-Clause for BLIP150K, COCO118K; Apache 2 for CC3M-Recap; ODC-BY-1.0 for Pixmo_Cap; see source materials for CC3M-Recap |
| OCR | SynthDog_EN, SynthDog_ZH, UReader, ART, COCO-Text, HierText, Uber-Text, TextOCR, OpenVINO, MLT-17 | 913K | Apache 2 for SynthDog_EN, SynthDog_ZH, UReader, TextOCR, OpenVINO; CC By 4.0 for COCO-Text; CC BY-SA 4.0 for HierText, Uber-Text; See source materials for ART, MLT-17 |
| Doc | DocVQA, DocStruct4M | 410K | Apache 2 |
| Table & Chart & Plot | Chart2Text, UniChart, PlotQA, WidgetCaption, Screen2Words, SciGraphQA-295K, Paper2Fig100K, MMC Instruction, M-Paper | 1.97M | GPL-3.0 for Chart2Text; MIT for UniChart, SciGraphQA-295K; Apache 2 for PlotQA, M-Paper; CC By 4.0 for WidgetCaption, Screen2Words, Paper2Fig100K; CC BY-SA 4.0 for MMC Instruction |
| Text Only | Evol-Instruct-GPT-4 | 70K | Apache 2 |
Instruction-tuning Stage
Further, to strengthen model’s understanding of science-based and general reasoning questions, as identified through error analysis, we oversampled (almost doubled the volume) specific datasets from the SFT dataset pool as detailed below.
Oversampled (~2x sampling rate): ScienceQA, AI2D, PMC-VQA, Cambrian, and TQA
Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.
For the details of training hyperparameters, please check our github repo
Contributors
Core contributors: Ximeng Sun, Aditya Kumar Singh, Gowtham Ramesh, Zicheng Liu
Contributors: Pratik Prabhanjan Brahma, Ze Wang, Jiang Liu, Jialian Wu, Prakamya Mishra, Xiaodong Yu, Yusheng Su, Sudhanshu Ranjan, Emad Barsoum
Bias, Risks, and Limitations
This model is made accessible without any safety guarantees. Users should be aware that the model may generate outputs that are sensitive, inaccurate, harmful, biased, or otherwise objectionable based on user prompts. It is crucial for users to conduct comprehensive safety evaluations, implement safety filtering, and verify the model's outputs to mitigate these risks.
License
See Files for license and any notices.
Citing
@misc{Instella-VL-1B,
title = {Instella-VL-1B: First AMD Vision Language Model},
url = {https://huggingface.co./amd/Instella-VL-1B},
author = {Ximeng Sun, Aditya Singh, Gowtham Ramesh, Jiang Liu, Ze Wang, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Prakamya Mishra, Jialian Wu, Xiaodong Yu, Yusheng Su, Emad Barsoum, Zicheng Liu},
month = {March},
year = {2025}
}
- Downloads last month
- 80
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no library tag.