Hugging Face's logo

OpenGVLab
/
InternVL2-8B

Image-Text-to-Text
Transformers
Safetensors
multilingual
internvl_chat
feature-extraction
internvl
vision
ocr
multi-image
video
custom_code
conversational
Model card Files Community
18
czczup commited on
Commit
ecae910
•
1 Parent(s): 581c716

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +66 -42
README.md CHANGED
@@ -9,6 +9,8 @@ pipeline_tag: image-text-to-text
9
 
10
  [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/706547971) \[🌟 [魔搭社区](https://modelscope.cn/organization/OpenGVLab) | [教程](https://mp.weixin.qq.com/s/OUaVLkxlk1zhFb1cvMCFjg) \]
11
 
 
 
12
  ## Introduction
13
 
14
  We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-8B model.
@@ -17,6 +19,16 @@ Compared to the state-of-the-art open-source multimodal large language models, I
17
 
18
  InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
19
 
 
 
 
 
 
 
 
 
 
 
20
  ## Model Details
21
 
22
  InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-8B consists of [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px), an MLP projector, and [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat).
@@ -25,27 +37,28 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
25
 
26
  ### Image Benchmarks
27
 
28
- | Benchmark | MiniCPM-Llama3-V-2_5 | InternVL-Chat-V1-5 | InternVL2-8B |
29
- | :--------------------------: | :------------------: | :----------------: | :----------: |
30
- | Model Size | 8.5B | 25.5B | 8.1B |
31
- | | | | |
32
- | DocVQA<sub>test</sub> | 84.8 | 90.9 | 91.6 |
33
- | ChartQA<sub>test</sub> | - | 83.8 | 83.3 |
34
- | InfoVQA<sub>test</sub> | - | 72.5 | 74.8 |
35
- | TextVQA<sub>val</sub> | 76.6 | 80.6 | 77.4 |
36
- | OCRBench | 725 | 724 | 794 |
37
- | MME<sub>sum</sub> | 2024.6 | 2187.8 | 2210.3 |
38
- | RealWorldQA | 63.5 | 66.0 | 64.4 |
39
- | AI2D<sub>test</sub> | 78.4 | 80.7 | 83.8 |
40
- | MMMU<sub>val</sub> | 45.8 | 45.2 / 46.8 | 49.3 / 51.2 |
41
- | MMBench-EN<sub>test</sub> | 77.2 | 82.2 | 81.7 |
42
- | MMBench-CN<sub>test</sub> | 74.2 | 82.0 | 81.2 |
43
- | CCBench<sub>dev</sub> | 45.9 | 69.8 | 75.9 |
44
- | MMVet<sub>GPT-4-0613</sub> | - | 62.8 | 60.0 |
45
- | MMVet<sub>GPT-4-Turbo</sub> | 52.8 | 55.4 | 54.2 |
46
- | SEED-Image | 72.3 | 76.0 | 76.2 |
47
- | HallBench<sub>avg</sub> | 42.4 | 49.3 | 45.2 |
48
- | MathVista<sub>testmini</sub> | 54.3 | 53.5 | 58.3 |
 
49
 
50
  - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
51
 
@@ -546,6 +559,16 @@ If you find this project useful in your research, please consider citing:
546
 
547
  InternVL 2.0 使用 8k 上下文窗口进行训练，训练数据包含长文本、多图和视频数据，与 InternVL 1.5 相比，其处理这些类型输入的能力显著提高。更多详细信息，请参阅我们的博客和 GitHub。
548
 
 
 
 
 
 
 
 
 
 
 
549
  ## 模型细节
550
 
551
  InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模型。对于每个规模的模型，我们都会发布针对多模态任务优化的指令微调模型。InternVL2-8B 包含 [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)、一个 MLP 投影器和 [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat)。
@@ -554,27 +577,28 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
554
 
555
  ### 图像相关评测
556
 
557
- | 评测数据集 | MiniCPM-Llama3-V-2_5 | InternVL-Chat-V1-5 | InternVL2-8B |
558
- | :--------------------------: | :------------------: | :----------------: | :----------: |
559
- | 模型大小 | 8.5B | 25.5B | 8.1B |
560
- | | | | |
561
- | DocVQA<sub>test</sub> | 84.8 | 90.9 | 91.6 |
562
- | ChartQA<sub>test</sub> | - | 83.8 | 83.3 |
563
- | InfoVQA<sub>test</sub> | - | 72.5 | 74.8 |
564
- | TextVQA<sub>val</sub> | 76.6 | 80.6 | 77.4 |
565
- | OCRBench | 725 | 724 | 794 |
566
- | MME<sub>sum</sub> | 2024.6 | 2187.8 | 2210.3 |
567
- | RealWorldQA | 63.5 | 66.0 | 64.4 |
568
- | AI2D<sub>test</sub> | 78.4 | 80.7 | 83.8 |
569
- | MMMU<sub>val</sub> | 45.8 | 45.2 / 46.8 | 49.3 / 51.2 |
570
- | MMBench-EN<sub>test</sub> | 77.2 | 82.2 | 81.7 |
571
- | MMBench-CN<sub>test</sub> | 74.2 | 82.0 | 81.2 |
572
- | CCBench<sub>dev</sub> | 45.9 | 69.8 | 75.9 |
573
- | MMVet<sub>GPT-4-0613</sub> | - | 62.8 | 60.0 |
574
- | MMVet<sub>GPT-4-Turbo</sub> | 52.8 | 55.4 | 54.2 |
575
- | SEED-Image | 72.3 | 76.0 | 76.2 |
576
- | HallBench<sub>avg</sub> | 42.4 | 49.3 | 45.2 |
577
- | MathVista<sub>testmini</sub> | 54.3 | 53.5 | 58.3 |
 
578
 
579
  - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
580
 
 
9
 
10
  [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 中文解读\]](https://zhuanlan.zhihu.com/p/706547971) \[🌟 [魔搭社区](https://modelscope.cn/organization/OpenGVLab) | [教程](https://mp.weixin.qq.com/s/OUaVLkxlk1zhFb1cvMCFjg) \]
11
 
12
+ [切换至中文版](#简介)
13
+
14
  ## Introduction
15
 
16
  We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-8B model.
 
19
 
20
  InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our blog and GitHub.
21
 
22
+ | Model Name | Vision Part | Language Part | HF Link | MS Link |
23
+ | :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |
24
+ | InternVL2-1B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B) |
25
+ | InternVL2-2B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B) |
26
+ | InternVL2-4B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B) |
27
+ | InternVL2-8B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B) |
28
+ | InternVL2-26B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B) |
29
+ | InternVL2-40B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B) |
30
+ | InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |
31
+
32
  ## Model Details
33
 
34
  InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-8B consists of [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px), an MLP projector, and [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat).
 
37
 
38
  ### Image Benchmarks
39
 
40
+ | Benchmark | MiniCPM-Llama3-V-2_5 | InternVL-Chat-V1-5 | InternVL2-8B |
41
+ | :-----------------------------: | :------------------: | :----------------: | :----------: |
42
+ | Model Size | 8.5B | 25.5B | 8.1B |
43
+ | | | | |
44
+ | DocVQA<sub>test</sub> | 84.8 | 90.9 | 91.6 |
45
+ | ChartQA<sub>test</sub> | - | 83.8 | 83.3 |
46
+ | InfoVQA<sub>test</sub> | - | 72.5 | 74.8 |
47
+ | TextVQA<sub>val</sub> | 76.6 | 80.6 | 77.4 |
48
+ | OCRBench | 725 | 724 | 794 |
49
+ | MME<sub>sum</sub> | 2024.6 | 2187.8 | 2210.3 |
50
+ | RealWorldQA | 63.5 | 66.0 | 64.4 |
51
+ | AI2D<sub>test</sub> | 78.4 | 80.7 | 83.8 |
52
+ | MMMU<sub>val</sub> | 45.8 | 45.2 / 46.8 | 49.3 / 51.2 |
53
+ | MMBench-EN<sub>test</sub> | 77.2 | 82.2 | 81.7 |
54
+ | MMBench-CN<sub>test</sub> | 74.2 | 82.0 | 81.2 |
55
+ | CCBench<sub>dev</sub> | 45.9 | 69.8 | 75.9 |
56
+ | MMVet<sub>GPT-4-0613</sub> | - | 62.8 | 60.0 |
57
+ | MMVet<sub>GPT-4-Turbo</sub> | 52.8 | 55.4 | 54.2 |
58
+ | SEED-Image | 72.3 | 76.0 | 76.2 |
59
+ | HallBench<sub>avg</sub> | 42.4 | 49.3 | 45.2 |
60
+ | MathVista<sub>testmini</sub> | 54.3 | 53.5 | 58.3 |
61
+ | OpenCompass<sub>avg-score</sub> | 58.8 | 61.7 | 64.1 |
62
 
63
  - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
64
 
 
559
 
560
  InternVL 2.0 使用 8k 上下文窗口进行训练，训练数据包含长文本、多图和视频数据，与 InternVL 1.5 相比，其处理这些类型输入的能力显著提高。更多详细信息，请参阅我们的博客和 GitHub。
561
 
562
+ | 模型名称 | 视觉部分 | 语言部分 | HF 链接 | MS 链接 |
563
+ | :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |
564
+ | InternVL2-1B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B) |
565
+ | InternVL2-2B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B) |
566
+ | InternVL2-4B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B) |
567
+ | InternVL2-8B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B) |
568
+ | InternVL2-26B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B) |
569
+ | InternVL2-40B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B) |
570
+ | InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |
571
+
572
  ## 模型细节
573
 
574
  InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模型。对于每个规模的模型，我们都会发布针对多模态任务优化的指令微调模型。InternVL2-8B 包含 [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)、一个 MLP 投影器和 [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat)。
 
577
 
578
  ### 图像相关评测
579
 
580
+ | 评测数据集 | MiniCPM-Llama3-V-2_5 | InternVL-Chat-V1-5 | InternVL2-8B |
581
+ | :-----------------------------: | :------------------: | :----------------: | :----------: |
582
+ | 模型大小 | 8.5B | 25.5B | 8.1B |
583
+ | | | | |
584
+ | DocVQA<sub>test</sub> | 84.8 | 90.9 | 91.6 |
585
+ | ChartQA<sub>test</sub> | - | 83.8 | 83.3 |
586
+ | InfoVQA<sub>test</sub> | - | 72.5 | 74.8 |
587
+ | TextVQA<sub>val</sub> | 76.6 | 80.6 | 77.4 |
588
+ | OCRBench | 725 | 724 | 794 |
589
+ | MME<sub>sum</sub> | 2024.6 | 2187.8 | 2210.3 |
590
+ | RealWorldQA | 63.5 | 66.0 | 64.4 |
591
+ | AI2D<sub>test</sub> | 78.4 | 80.7 | 83.8 |
592
+ | MMMU<sub>val</sub> | 45.8 | 45.2 / 46.8 | 49.3 / 51.2 |
593
+ | MMBench-EN<sub>test</sub> | 77.2 | 82.2 | 81.7 |
594
+ | MMBench-CN<sub>test</sub> | 74.2 | 82.0 | 81.2 |
595
+ | CCBench<sub>dev</sub> | 45.9 | 69.8 | 75.9 |
596
+ | MMVet<sub>GPT-4-0613</sub> | - | 62.8 | 60.0 |
597
+ | MMVet<sub>GPT-4-Turbo</sub> | 52.8 | 55.4 | 54.2 |
598
+ | SEED-Image | 72.3 | 76.0 | 76.2 |
599
+ | HallBench<sub>avg</sub> | 42.4 | 49.3 | 45.2 |
600
+ | MathVista<sub>testmini</sub> | 54.3 | 53.5 | 58.3 |
601
+ | OpenCompass<sub>avg-score</sub> | 58.8 | 61.7 | 64.1 |
602
 
603
  - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。MMMU、OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
604