Reverb/Idefics2-8b-docVQA-finetuned

Idefics2

Idefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text outputs. The model can answer questions about images, describe visual content, create stories grounded on multiple images, or simply behave as a pure language model without visual inputs.

Model Information

• Base Model: HuggingFaceM4/idefics2-8b
• Dataset Used: DocVQA dataset
  • Introduced in Mathew et al. (2021)
  • Consists of 50,000 questions defined on 12,000+ document images
  • For further information, visit the challenge page and paper

Training Details

• The training process took approximately 38hours on an A100 80GB GPU, and model was fine-tuned using QLoRA.
• Trained with 39.5k train dataset from DocVQA single page questions
• Training Log:

Epoch	Loss	Grad Norm	Learning Rate
0.01	2.3776	10.40	4.8e-05
0.25	0.5029	6.10	9.5412e-05
0.50	0.434	5.74	7.5973e-05
0.75	0.4608	7.46	7.3925e-05
1.0	0.3846	4.77	5.0369e-05
1.25	0.3226	3.63	4.9857e-05
1.5	0.3175	5.03	2.5277e-05
1.75	0.2918	5.63	2.5789e-05

{'train_runtime': 141781.6786, 'train_samples_per_second': 0.557, 'train_steps_per_second': 0.035, 'train_loss': 0.3973848872424526, 'epoch': 2.0}

Technical summary

Idefics2 exhibits strong performance for a model of its size (8B parameters) when compared to other open multimodal models and is often competitive with closed-source systems. As such, it serves as a strong foundation for various use-case specific fine-tunings.

For more details, expand the result table.

Model	Open weights	Size	# tokens per image	MMMU (val/test)	MathVista (testmini)	TextVQA (val)	MMBench (test)	VQAv2 (test-dev)	DocVQA (test)
DeepSeek-VL	✅	7B	576	36.6/-	36.1	64.4	73.2	-	49.6
LLaVa-NeXT-Mistral-7B	✅	7B	2880	35.3/-	37.7	65.7	68.7	82.2	-
LLaVa-NeXT-13B	✅	13B	2880	36.2/-	35.3	67.1	70.0	82.8	-
LLaVa-NeXT-34B	✅	34B	2880	51.1/44.7	46.5	69.5	79.3	83.7	-
MM1-Chat-7B	❌	7B	720	37.0/35.6	35.9	72.8	72.3	-	-
MM1-Chat-30B	❌	30B	720	44.7/40.3	39.4	73.5	75.1	83.7
Gemini 1.0 Pro	❌	🤷‍♂️	🤷‍♂️	47.9/-	45.2	74.6	-	71.2	88.1
Gemini 1.5 Pro	❌	🤷‍♂️	🤷‍♂️	58.5/-	52.1	73.5	-	73.2	86.5
Claude 3 Haiku	❌	🤷‍♂️	🤷‍♂️	50.2/-	46.4	-	-	-	88.8
Idefics1 instruct (32-shots)	✅	80B	-	-	-	39.3	-	68.8	-
Idefics2 (w/o im. split)	✅	8B	64	43.5/37.9	51.6	70.4	76.8	80.8	67.3
Idefics2 (w/ im. split)	✅	8B	320	43.0/37.7	51.4	73.0	76.7	81.2	74.0
Idefics2 DocVQA Finetuned (w/ im. split)	✅	8B	320	43.0/37.7	52.5	72.0	77.7	81.1	72.5

Idefics2 is trained in 2 stages for maximum efficiency. In a first stage, images are fed to the model at SigLIP's native resolution (squares of 384 x 384). In the second stage, images are fed to the model at their native resolution (with a maximum of 980 and a minimum of 378) and native aspect ratio.

We use DoRA to train the parameters initialized from pre-trained backbones and full fine-tuning for newly initialized parameters (modality connector), as we find this strategy to be more stable as well as more computationally efficient.

Vision Encoder Efficiency

Given the high resolution supported, the vision part of the model can be memory hungry depending on your configuration. If you are GPU-memory-constrained, you can:

1. Deactivate image splitting: To do so, add do_image_splitting=False when initializing the processor (AutoProcessor.from_pretrained). There are no changes required on the model side. Note that only the SFT model has been trained with image splitting.

2. Decrease maximum image resolution: To do so, add size={"longest_edge": 448, "shortest_edge": 378} when initializing the processor (AutoProcessor.from_pretrained). In particular, the longest_edge value can be adapted to fit the need (the default value is 980). We recommend using values that are multiples of 14. There are no changes required on the model side.

do_image_splitting=True is especially needed to boost performance on OCR tasks where a very large image is used as input. For regular VQA or captioning tasks, this argument can be safely set to False with minimal impact on performance (see the evaluation table above).

How to Get Started

This section shows snippets of code for generation for idefics2-8b-DocVQA-finetuned. Let's first define some common imports and inputs.

import requests
import torch
from PIL import Image
from io import BytesIO

from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image

DEVICE = "cuda:0"

# Note that passing the image urls (instead of the actual pil images) to the processor is also possible
image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")

For idefics2-8b-base

Click to expand.

processor = AutoProcessor.from_pretrained("idefics2-8b-DocVQA-finetuned")
model = AutoModelForVision2Seq.from_pretrained(
    "Reverb/idefics2-8b-DocVQA-finetuned",
).to(DEVICE)

# Create inputs
prompts = [
  "<image>In this image, we can see the city of New York, and more specifically the Statue of Liberty.<image>In this image,",
  "In which city is that bridge located?<image>",
]
images = [[image1, image2], [image3]]
inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)
# ['In this image, we can see the city of New York, and more specifically the Statue of Liberty. In this image, we can see the city of Chicago, and more specifically the skyscrapers of the city.', 'In which city is that bridge located? The Golden Gate Bridge is a suspension bridge spanning the Golden Gate, the one-mile-wide (1.6 km) strait connecting San Francisco Bay and the Pacific Ocean. The structure links the American city of San Francisco, California — the northern tip of the San Francisco Peninsula — to Marin County, carrying both U.S. Route 101 and California State Route 1 across the strait. The bridge is one of the most internationally recognized symbols of San Francisco, California, and the United States. It has been declared one of the Wonders of the Modern World by the American Society of Civil Engineers.\n\nThe Golden Gate Bridge is a suspension bridge spanning the Golden Gate, the one-mile-wide (1.6 km) strait connecting San Francisco Bay and the Pacific Ocean. The structure links the American city of San Francisco, California — the northern tip of the San Francisco Peninsula — to Marin County, carrying both U.S. Route 101 and California State Route 1 across the strait. The bridge is one of the most internationally recognized symbols of San Francisco, California, and the United States. It has been declared one of the Wonders of the Modern World by the American Society of Civil Engineers.\n\nThe Golden Gate Bridge is a suspension bridge spanning the Golden Gate, the one-mile-wide (1.6 km) strait connecting San Francisco Bay and the Pacific Ocean. The structure links the American city of San Francisco, California — the northern tip of the San Francisco Peninsula — to Marin County, carrying both U.S. Route 101 and California State Route 1 across the strait. The bridge is one of the most internationally recognized symbols of San Francisco, California, and the United States. It has been declared one of the Wonders of the Modern World by the American Society of Civil Engineers.\n\nThe Golden Gate Bridge is a suspension bridge spanning the Golden Gate, the one-mile-wide (1.6 km) strait connecting San Francisco Bay and the Pacific Ocean. The structure links the American city of San Francisco, California — the northern tip of the San Francisco Peninsula — to Marin County, carrying both U.S. Route 101 and California State Route 1 across the strait. The bridge is one of the most internationally recognized symbols of San Francisco, California, and']

Text generation inference

Idefics2 is integrated into TGI.

Multiple images can be passed on with the markdown syntax (![](IMAGE_URL)) and no spaces are required before and after. The dialogue utterances can be separated with <end_of_utterance>\n followed by User: or Assistant:. User: is followed by a space if the following characters are real text (no space if followed by an image).

Model optimizations

If your GPU allows, we first recommend loading (and running inference) in half precision (torch.float16 or torch.bfloat16).

model = AutoModelForVision2Seq.from_pretrained(
    "Reverb/idefics2-8b-DocVQA-finetuned",
+    torch_dtype=torch.float16,    
).to(DEVICE)

Vision encoder efficiency

Given the high resolution supported, the vision part of the model can be memory hungry depending on your configuration. If you are GPU-memory-constrained, you can:

• deactivate the image splitting. To do so, add do_image_splitting=False when initializing the processor (AutoProcessor.from_pretrained). There are no changes required on the model side. Note that only the sft model has been trained with image splitting.
• decrease the maximum image resolution. To do so, add size= {"longest_edge": 448, "shortest_edge": 378} when initializing the processor (AutoProcessor.from_pretrained). In particular, the longest_edge value can be adapted to fit the need (the default value is 980). We recommend using values that are multiples of 14. There are no changes required on the model side.

do_image_splitting=True is especially needed to boost performance on OCR tasks where a very large image is used as input. For the regular VQA or captioning tasks, this argument can be safely set to False with minimal impact on performance (see the evaluation table above).

Using Flash-attention 2 to speed up generation

Click to expand.

First, make sure to install flash-attn. Refer to the original repository of Flash Attention for the package installation. Simply change the snippet above with:

model = AutoModelForVision2Seq.from_pretrained(
    "Reverb/idefics2-8b-DocVQA-finetuned",
+    torch_dtype=torch.float16,    
+    _attn_implementation="flash_attention_2",
).to(DEVICE)

These optimizations can be combined to suit variable trade-offs between GPU memory, inference speed and performance. We provide the following comparison as anchor points to guide the user in choosing necessary optimizations. All of these benchmarks were computed with the example code snippet described above on a H100 (see colab). As one can see, the are a few setups that require less than 24GB of GPU memory.

Flash attention 2	Image splitting	Float type	4 bits quantization	Peak GPU memory (GB)	Time for 20 generations (secs)
No	Yes	fp32	No	54.9	55.6
No	Yes	bf16	No	41.3	34.3
No	Yes	fp16	No	36.7	33.3
Yes	Yes	fp16	No	21.0	13.3
Yes	Yes	fp16	bitsandbytes (entire model)	8.9	19.9
No	Yes	fp16	bitsandbytes (entire model)	24.7	40.4
No	Yes	fp16	AWQ (LLM only)	26.4	37.1
Yes	Yes	fp16	AWQ (LLM only)	10.7	16.3
No	Yes	fp16	AWQ + fusing (LLM only)	26.0	38.4
No	No	fp32	No	38.8	17.5
No	No	bf16	No	22.2	14.4
No	No	fp16	No	21.3	13.9
Yes	No	fp16	No	18.1	10.4
Yes	No	fp16	bitsandbytes (entire model)	6.0	17.3
No	No	fp16	bitsandbytes (entire model)	9.2	20.9
No	No	fp16	AWQ (LLM only)	10.9	15.9
Yes	No	fp16	AWQ (LLM only)	7.8	12.3
No	No	fp16	AWQ + fusing (LLM only)	10.5	19.5

To learn more quantization schemes and fusing, we refer to the documentation.

Bias, Risks, and Limitations

• The model currently will offer medical diagnosis when prompted to do so (vqa-rad, a dataset of QA pairs on radiology images is present in the SFT mixture). For example, the prompt Does this X-ray show any medical problems? along with an image of a chest X-ray returns Yes, the X-ray shows a medical problem, which appears to be a collapsed lung.. We discourage users from using the model on medical applications without proper adaptation and evaluation.
• Despite our efforts in filtering the training data, we found a small proportion of content that is not suitable for all audiences. This includes pornographic content and reports of violent shootings and is prevalent in the OBELICS portion of the data (see here for more details). As such, the model is susceptible to generating text that resembles this content.
• We note that we know relatively little about the composition of the pre-trained LM backbone, which makes it difficult to link inherited limitations or problematic behaviors to their data.

Misuse and Out-of-scope use

Using the model in high-stakes settings is out of scope for this model. The model is not designed for critical decisions nor uses with any material consequences on an individual's livelihood or wellbeing. The model outputs content that appears factual but may not be correct. Out-of-scope uses include:

• Usage for evaluating or scoring individuals, such as for employment, education, or credit
• Applying the model for critical automatic decisions, generating factual content, creating reliable summaries, or generating predictions that must be correct

Intentionally using the model for harm, violating human rights, or other kinds of malicious activities, is a misuse of this model. This includes:

• Spam generation
• Disinformation and influence operations
• Disparagement and defamation
• Harassment and abuse
• Deception
• Unconsented impersonation and imitation
• Unconsented surveillance

Citation

BibTeX:

@misc{laurencon2023obelics,
      title={OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents},
      author={Hugo Laurençon and Lucile Saulnier and Léo Tronchon and Stas Bekman and Amanpreet Singh and Anton Lozhkov and Thomas Wang and Siddharth Karamcheti and Alexander M. Rush and Douwe Kiela and Matthieu Cord and Victor Sanh},
      year={2023},
      eprint={2306.16527},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}

@misc{laurençon2024matters,
      title={What matters when building vision-language models?}, 
      author={Hugo Laurençon and Léo Tronchon and Matthieu Cord and Victor Sanh},
      year={2024},
      eprint={2405.02246},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}