File size: 7,384 Bytes
7e38877 12a0d79 928abca 9c7e52c d9e2106 45b5767 9c7e52c 954f9f4 9c7e52c 69c10d4 9c7e52c a0d7a21 9c7e52c 69c10d4 40c6b18 9c7e52c 37287d1 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 da411b9 69c10d4 9c7e52c 954f9f4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 |
---
license: apache-2.0
datasets:
- liuhaotian/LLaVA-Pretrain
- liuhaotian/LLaVA-Instruct-150K
pipeline_tag: visual-question-answering
---
## Model
llava-dinov2-internlm2-7b-v1 is a LLaVA model fine-tuned from [InternLM2-Chat-7B](https://huggingface.co./internlm/internlm2-chat-7b) and [Dinov2-large](https://huggingface.co./facebook/dinov2-large) with [LLaVA-Pretrain](liuhaotian/LLaVA-Pretrain) and [LLaVA-Instruct](https://huggingface.co./datasets/liuhaotian/LLaVA-Instruct-150K) by [XTuner](https://github.com/InternLM/xtuner). I thank the help of [Zhihao Lin](https://github.com/LZHgrla) and [pppppM](https://github.com/pppppM) from the Xtuner team. I also thank the Huggingface transformers team for approving [my pull request](https://github.com/huggingface/transformers/pull/28504) so training Dinov2 in bf16 becomes possible.
I did not carefully tune the training hyperparameters but the model still show capability to solve some tasks. It shows that a visual encoder can be integrated with an LLM, even when the encoder is not aligned with natural language with contrastive learning like CLIP.
## Example
Explain the photo in English:
Explain the photo in Chinese:
## Rank
## Results
Model | MMBench Test (EN) | MMBench Dev (EN) | MMBench Test (CN) | MMBench Dev (CN) | CCBench Dev
------------- | ------------- | ------------- | ------------- | ------------- | -------------
LLaVA-v1.5-7B | 67.7 | 69.2 | 61.0 | 59.7 | 28.4
LLaVA-InternLM-7B | 69.0 | 68.5 | 66.7 | 63.8 | 37.3
LLaVA-InternLM2-7B | 73.3 | 74.6 | 71.7 | 72.0 | 42.5
llava-dinov2-internlm2-7b-v1 | 64.0 | 65.2 | 62.9 | 61.6 | 45.3
## Installation
```
git clone https://github.com/InternLM/xtuner
pip install -e ./xtuner[deepspeed]
cd ./xtuner
# Now replace the source code files with the modifed version in modified_xtuner_code directory
```
## Chat
```
xtuner chat internlm/internlm2-chat-7b \
--visual-encoder facebook/dinov2-large\
--llava ./lora_and_projector \
--prompt-template internlm2_chat \
--image $IMAGE_PATH
```
## Common Errors
1.
```
command error: 'libGL.so.1: cannot open shared object file: No such file or directory'!
```
You can solve it by
```
# For Ubuntu
sudo apt-get update
sudo apt-get install libgl1-mesa-glx
# For CentOS and Fedora
sudo yum install mesa-libGL
```
2.
```
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
```
You can solve it by reinstall numpy.
3.
```
ImportError:
InternLM2Converter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
```
You just need
```
pip install protobuf
```
## Data prepration
1. File structure
```
./data/llava_data
├── LLaVA-Pretrain
│ ├── blip_laion_cc_sbu_558k.json
│ ├── blip_laion_cc_sbu_558k_meta.json
│ └── images
├── LLaVA-Instruct-150K
│ └── llava_v1_5_mix665k.json
└── llava_images
├── coco
│ └── train2017
├── gqa
│ └── images
├── ocr_vqa
│ └── images
├── textvqa
│ └── train_images
└── vg
├── VG_100K
└── VG_100K_2
```
2. Pretrain Data
LLaVA-Pretrain
```shell
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co./datasets/liuhaotian/LLaVA-Pretrain --depth=1
```
3. Finetune Data
3.1 Text data
LLaVA-Instruct-150K
```shell
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co./datasets/liuhaotian/LLaVA-Instruct-150K --depth=1
```
3.2 Image data
3.2.1 COCO (coco): [train2017](http://images.cocodataset.org/zips/train2017.zip)
3.2.2 GQA (gqa): [images](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)
3.2.3 OCR-VQA (ocr_vqa): [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing)
⚠️⚠️⚠️ Modify the name of OCR-VQA's images to keep the extension as `.jpg`!
```shell
#!/bin/bash
ocr_vqa_path="<your-directory-path>"
find "$target_dir" -type f | while read file; do
extension="${file##*.}"
if [ "$extension" != "jpg" ]
then
cp -- "$file" "${file%.*}.jpg"
fi
done
```
3.2.4 TextVQA (textvqa): [train_val_images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip)
3.2.5 VisualGenome (VG): [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip)
## Cheers! Now train your own model!
1. Alignment module pretraining
```
NPROC_PER_NODE=8 xtuner train ./llava_internlm2_chat_7b_dinov2_e1_gpu8_pretrain.py --deepspeed deepspeed_zero2
```
The checkpoint and tensorboard logs are saved by default in ./work_dirs/. I only train it for 1 epoch to be same as the original LLaVA paper. Some researches also report that training for multiple epochs will make the model overfit the training dataset and perform worse in other domains.
Here is my loss curve:
2. Instruction following fine-tuning
```
NPROC_PER_NODE=8 xtuner train ./llava_internlm2_chat_7b_dinov2_e1_gpu8_finetune.py --deepspeed deepspeed_zero2
```
Here is my loss curve (the curve fluctuates strongly because the batch size is small, and I only record batch loss instead of epoch loss):
## Transfer the checkpoints to Huggingface safetensor format
```
xtuner convert pth_to_hf ./llava_internlm2_chat_7b_dinov2_e1_gpu8_finetune.py ./work_dirs/epoch_1.pth ./my_lora_and_projector
```
The adapter still need to be used with the internlm/internlm2-chat-7b and facebook/dinov2-large models. I have not tried to merge them yet but it is possible with Xtuner, see this [tutorial](https://github.com/InternLM/xtuner/blob/f63859b3d0cb39cbac709e3850f3fe01de1023aa/xtuner/configs/llava/README.md#L4).
## MMBench Evaluation
```
xtuner mmbench internlm/internlm2-chat-7b \
--visual-encoder facebook/dinov2-large \
--llava ./my_lora_and_projector \
--prompt-template internlm2_chat \
--data-path $MMBENCH_DATA_PATH \
--work-dir $RESULT_PATH
```
## Deployment
Xtuner team is developing HF chatbot (based on Huggingface transformers) and LMDeploy chatbot (based on TurboMind). I am waiting for their final version of API. |