add model card (#2)

- add model card (d112364f9360c26972d07217a9fd6b7219ae62e1)


Co-authored-by: Aritra Roy Gosthipaty <[email protected]>

README.md

@@ -1,202 +1,172 @@
 ---
 library_name: transformers
-tags: []
+license: apache-2.0
+datasets:
+- HuggingFaceM4/the_cauldron
+- HuggingFaceM4/Docmatix
+pipeline_tag: image-text-to-text
+language:
+- en
 base_model:
 - HuggingFaceTB/SmolLM2-1.7B-Instruct
 - google/siglip-so400m-patch14-384
 ---
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/SmolVLM.png" width="800" height="auto" alt="Image description">
 
-# Model Card for Model ID
+# SmolVLM
 
-<!-- Provide a quick summary of what the model is/does. -->
+SmolVLM is a compact open multimodal model that accepts arbitrary sequences of image and text inputs to produce text outputs.
+Designed for efficiency, SmolVLM can answer questions about images, describe visual content, create stories grounded on multiple images,
+or function as a pure language model without visual inputs. Its lightweight architecture makes it suitable for on-device applications
+while maintaining strong performance on multimodal tasks.
 
+## Model Summary
 
+- **Developed by:** Hugging Face 🤗
+- **Model type:** Multi-modal model (image+text)
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Architecture:** Based on [Idefics3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) (see technical summary)
 
-## Model Details
+## Resources
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Demo:** [SmolVLM Demo](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM)
+- **Blog:** [Blog post](https://huggingface.co/blog/smolvlm)
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
+SmolVLM can be used for inference on multimodal (image + text) tasks where the input comprises text queries along with one or more images.
+Text and images can be interleaved arbitrarily, enabling tasks like image captioning, visual question answering, and storytelling based on
+visual content. The model does not support image generation.
+
+To fine-tune SmolVLM on a specific task, you can follow the fine-tuning tutorial.
+<!-- todo: add link to fine-tuning tutorial -->
+
+### Technical Summary
+
+SmolVLM leverages the lightweight SmolLM2 language model to provide a compact yet powerful multimodal experience.
+It introduces several changes compared to previous Idefics models:
+
+- **Image compression:** We introduce a more radical image compression compared to Idefics3 to enable the model to infer faster and use less RAM.
+- **Visual Token Encoding:** SmolVLM uses 81 visual tokens to encode image patches of size 384×384. Larger images are divided into patches, each encoded separately, enhancing efficiency without compromising performance.
+
+More details about the training and architecture are available in our technical report.
+
+
+### How to get started
+
+You can use transformers to load, infer and fine-tune SmolVLM.
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoProcessor, AutoModelForVision2Seq
+from transformers.image_utils import load_image
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+# Load images
+image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
+image2 = load_image("https://huggingface.co/spaces/merve/chameleon-7b/resolve/main/bee.jpg")
+# Initialize processor and model
+processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM-Synthetic")
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-Synthetic",
+    torch_dtype=torch.bfloat16,
+    _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "eager",
+).to(DEVICE)
+# Create input messages
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image"},
+            {"type": "image"},
+            {"type": "text", "text": "Can you describe the two images?"}
+        ]
+    },
+]
+# Prepare inputs
+prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
+inputs = processor(text=prompt, images=[image1, image2], return_tensors="pt")
+inputs = inputs.to(DEVICE)
+# Generate outputs
+generated_ids = model.generate(**inputs, max_new_tokens=500)
+generated_texts = processor.batch_decode(
+    generated_ids,
+    skip_special_tokens=True,
+)
+print(generated_texts[0])
+"""
+User:<image>Can you describe the two images?
+Assistant: The two images are not described in the provided facts, so we cannot provide any information about them.
+"""
+```
+
+
+### Model optimizations
+
+**Precision**: For better performance, load and run the model in half-precision (`torch.float16` or `torch.bfloat16`) if your hardware supports it.
+
+```python
+from transformers import AutoModelForVision2Seq
+import torch
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-Synthetic",
+    torch_dtype=torch.bfloat16
+).to("cuda")
+```
+
+You can also load SmolVLM with 4/8-bit quantization using bitsandbytes, torchao or Quanto. Refer to [this page](https://huggingface.co/docs/transformers/en/main_classes/quantization) for other options.
+
+```python
+from transformers import AutoModelForVision2Seq, BitsAndBytesConfig
+import torch
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-Synthetic",
+    quantization_config=quantization_config,
+)
+```
+
+**Vision Encoder Efficiency**: Adjust the image resolution by setting `size={"longest_edge": N*384}` when initializing the processor, where N is your desired value. The default `N=4` works well, which results in input images of
+size 1536×1536. For documents, `N=5` might be beneficial. Decreasing N can save GPU memory and is appropriate for lower-resolution images. This is also useful if you want to fine-tune on videos.
+
+
+## Misuse and Out-of-scope Use
+
+SmolVLM is not intended for high-stakes scenarios or critical decision-making processes that affect an individual's well-being or livelihood. The model may produce content that appears factual but may not be accurate. Misuse includes, but is not limited to:
+
+- Prohibited Uses:
+  - Evaluating or scoring individuals (e.g., in employment, education, credit)
+  - Critical automated decision-making
+  - Generating unreliable factual content
+- Malicious Activities:
+  - Spam generation
+  - Disinformation campaigns
+  - Harassment or abuse
+  - Unauthorized surveillance
+
+### License
+
+SmolVLM is built upon [the shape-optimized SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384) as image encoder and [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) for text decoder part.
+
+We release the SmolVLM checkpoints under the Apache 2.0 license.
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
+The training data comes from [The Cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) and [Docmatix](https://huggingface.co/datasets/HuggingFaceM4/Docmatix) datasets, with emphasis on document understanding (25%) and image captioning (18%), while maintaining balanced coverage across other crucial capabilities like visual reasoning, chart comprehension, and general instruction following.
+<img src="https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct/resolve/main/mixture_the_cauldron.png" alt="Example Image" style="width:90%;" />
 
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+| Model             | MMMU (val) | MathVista (testmini) | MMStar (val) | DocVQA (test) | TextVQA (val) | Min GPU RAM required (GB) |
+|-------------------|------------|----------------------|--------------|---------------|---------------|---------------------------|
+| SmolVLM           | 38.8       | 44.6                | 42.1         | 81.6          | 72.7          | 5.02                      |
+| Qwen-VL 2B        | 41.1       | 47.8                | 47.5         | 90.1          | 79.7          | 13.70                     |
+| InternVL2 2B      | 34.3       | 46.3                | 49.8         | 86.9          | 73.4          | 10.52                     |
+| PaliGemma 3B 448px| 34.9       | 28.7                | 48.3         | 32.2          | 56.0          | 6.72                      |
+| moondream2        | 32.4       | 24.3                | 40.3         | 70.5          | 65.2          | 3.87                      |
+| MiniCPM-V-2       | 38.2       | 39.8                | 39.1         | 71.9          | 74.1          | 7.88                      |
+| MM1.5 1B          | 35.8       | 37.2                | 0.0          | 81.0          | 72.5          | NaN                       |