init commit

README.md

@@ -0,0 +1,166 @@
+---
+language: en
+datasets:
+- laion2b
+---
+
+# OpenFlamingo-3B (CLIP ViT-L/14, MPT-1B)
+
+[Blog post]() | [Code](https://github.com/mlfoundations/open_flamingo) | [Demo]()
+
+OpenFlamingo is an open source implementation of DeepMind's [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model) models. 
+This 3B-parameter model uses a [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14) vision encoder and [MPT-1B](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b) language model.
+
+## Model Details
+We follow the Flamingo modeling paradigm, outfitting the layers of a pretrained, frozen language model such that they cross-attend to visual features when decoding. Following Flamingo, we freeze the vision encoder and language model but train the connecting modules on web-scraped image-text sequences. Specifically, we trained this model on a mixture of [LAION-2B](https://arxiv.org/abs/2210.08402) and [Multimodal C4](https://arxiv.org/abs/2304.06939).  
+
+This model has cross-attention modules inserted in *every* decoder block. It was trained using DistributedDataParallel across 64 A100 80GB GPUs at FP32 precision.
+
+The [MPT-1B](https://huggingface.co/mosaicml/mpt-1b-redpajama-200b) modeling code does not accept the `labels` kwarg and compute cross-entropy loss within `forward()`. To train with the OpenFlamingo codebase, we suggest a version with the `labels` kwarg [here](https://huggingface.co/anas-awadalla/mpt-1b-redpajama-200b).
+
+## Uses
+OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification. 
+
+### Generation example
+Below is an example of generating text conditioned on interleaved images/text. In particular, let's try few-shot image captioning.
+
+``` python
+from PIL import Image
+import requests
+
+"""
+Step 1: Load images
+"""
+demo_image_one = Image.open(
+    requests.get(
+        "http://images.cocodataset.org/val2017/000000039769.jpg", stream=True
+    ).raw
+)
+
+demo_image_two = Image.open(
+    requests.get(
+        "http://images.cocodataset.org/test-stuff2017/000000028137.jpg",
+        stream=True
+    ).raw
+)
+
+query_image = Image.open(
+    requests.get(
+        "http://images.cocodataset.org/test-stuff2017/000000028352.jpg", 
+        stream=True
+    ).raw
+)
+
+
+"""
+Step 2: Preprocessing images
+Details: For OpenFlamingo, we expect the image to be a torch tensor of shape 
+ batch_size x num_media x num_frames x channels x height x width. 
+ In this case batch_size = 1, num_media = 3, num_frames = 1,
+ channels = 3, height = 224, width = 224.
+"""
+vision_x = [image_processor(demo_image_one).unsqueeze(0), image_processor(demo_image_two).unsqueeze(0), image_processor(query_image).unsqueeze(0)]
+vision_x = torch.cat(vision_x, dim=0)
+vision_x = vision_x.unsqueeze(1).unsqueeze(0)
+
+"""
+Step 3: Preprocessing text
+Details: In the text we expect an <image> special token to indicate where an image is.
+ We also expect an <|endofchunk|> special token to indicate the end of the text 
+ portion associated with an image.
+"""
+tokenizer.padding_side = "left" # For generation padding tokens should be on the left
+lang_x = tokenizer(
+    ["<image>An image of two cats.<|endofchunk|><image>An image of a bathroom sink.<|endofchunk|><image>An image of"],
+    return_tensors="pt",
+)
+
+
+"""
+Step 4: Generate text
+"""
+generated_text = model.generate(
+    vision_x=vision_x,
+    lang_x=lang_x["input_ids"],
+    attention_mask=lang_x["attention_mask"],
+    max_new_tokens=20,
+    num_beams=3,
+)
+
+print("Generated text: ", tokenizer.decode(generated_text[0]))
+```
+
+### Bias, Risks, and Limitations
+OpenFlamingo models inherit the risks of their parent models, especially the language model. As an open-source research effort, we highly value open, accessible, reproducible multimodal model research; however, it is crucial to be aware that these models are trained on web data, have not been finetuned for safety, and thus may produce unintended, inappropriate, unreliable, and/or inaccurate outputs. Please use caution before deploying OpenFlamingo models in real applications. We also hope that OpenFlamingo enables further safety and reliability research to address these issues.
+
+In an effort to mitigate current potential biases and harms, we have deployed a text content filter on model outputs in the OpenFlamingo demo. We continue to red-team the model to understand and improve its safety.
+
+## Evaluation
+
+<table>
+  <tr>
+    <th></th>
+    <th>0-shot</th>
+    <th>4-shot</th>
+    <th>8-shot</th>
+    <th>16-shot</th>
+    <th>32-shot</th>
+  </tr>
+  <tr>
+    <th>COCO (CIDEr)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+  </tr>
+  <tr>
+    <th>Flickr-30K (CIDEr)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+  </tr>
+  <tr>
+    <th>VQAv2 (Accuracy)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+  </tr>
+  <tr>
+    <th>OK-VQA (Accuracy)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+  </tr>
+  <tr>
+    <th>TextVQA (Accuracy)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+    
+  </tr>
+  <tr>
+    <th>Vizwiz (Accuracy)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+  </tr>  
+  <tr>
+    <th>Hateful Memes (ROC AUC)</th>
+    <td>0</td>
+    <td>4</td>
+    <td>8</td>
+    <td>16</td>
+    <td>32</td>
+  </tr>
+</table>