File size: 4,471 Bytes
593a09f da0e35f 593a09f bebf664 60dba5d bebf664 f0164d3 bebf664 598ad31 df457f7 25b4203 bebf664 eb37f82 598ad31 eb37f82 598ad31 bebf664 6f3de67 bebf664 9a775d9 bebf664 25b4203 bebf664 25b4203 709aa5b 25b4203 bebf664 9a775d9 bebf664 9a775d9 bebf664 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 |
---
license: apache-2.0
tags:
- llava
pipeline_tag: image-text-to-text
---
## Model
llava-clip-internlm2-1_8b-pretrain-v1 is a LLaVA checkpoint finetuned from [internlm2-chat-1_8b](https://huggingface.co./internlm/internlm2-chat-1_8b) and [CLIP-ViT-Large-patch14-336](https://huggingface.co./openai/clip-vit-large-patch14-336) with [LLaVA-Pretrain](liuhaotian/LLaVA-Pretrain) by [Xtuner](https://github.com/InternLM/xtuner). The pretraining phase took 16 hours on a single Nvidia A6000 ada GPU.
The total size of the model is around 2.2B, which is suitable for embedded applications like robotics.
#### I just finished the pretrain phase of the model. I will release the full finetuned model soon. You can also finetune your own version based on the checkpoint here.
## Installation
```
git clone https://github.com/InternLM/xtuner
pip install -e ./xtuner[deepspeed]
apt install git-lfs
git clone https://huggingface.co./StarCycle/llava-clip-internlm2-1_8b-pretrain-v1
cd ./llava-clip-internlm2-1_8b-pretrain-v1
```
## Common Errors
1.
```
command error: 'libGL.so.1: cannot open shared object file: No such file or directory'!
```
You can solve it by
```
# For Ubuntu
sudo apt-get update
sudo apt-get install libgl1-mesa-glx
# For CentOS and Fedora
sudo yum install mesa-libGL
```
2.
```
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
```
You can solve it by reinstall numpy.
3.
```
ImportError:
InternLM2Converter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
```
You just need
```
pip install protobuf
```
4.
To use tensorboard to visualize the training loss curve:
```
pip install future tensorboard
```
5. If your training process is killed during data preprocessing, you can modify the `map_num_proc` in xtuner/xtuner/dataset
/huggingface.py
```
def process(dataset,
do_dataset_tokenization=True,
tokenizer=None,
max_length=None,
dataset_map_fn=None,
template_map_fn=None,
max_dataset_length=None,
split='train',
remove_unused_columns=False,
rename_maps=[],
shuffle_before_pack=True,
pack_to_max_length=True,
use_varlen_attn=False,
input_ids_with_output=True,
with_image_token=False,
map_num_proc=32): # modify it to a smaller number, e.g., 4
```
6. If you fail to load the model, check whether you installed git-lfs and actually downloaded the model file.
## Data prepration
1. File structure
```
# . means the llava-clip-internlm2-1_8b-pretrain-v1 folder you clone
./data/llava_data
├── LLaVA-Pretrain
  ├── blip_laion_cc_sbu_558k.json
  ├── blip_laion_cc_sbu_558k_meta.json
  └── images
```
2. Pretrain Data
LLaVA-Pretrain
```shell
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co./datasets/liuhaotian/LLaVA-Pretrain --depth=1
```
3. Finetune Data
Please check the final release version
## Cheers! Now train your own model!
1. Alignment module pretraining
```
# single GPU
xtuner train ./llava_internlm2_chat_1_8b_clip_vit_large_p14_336_e1_gpu1_pretrain.py --deepspeed deepspeed_zero2
# multiple GPU
NPROC_PER_NODE=8 xtuner train ./llava_internlm2_chat_1_8b_clip_vit_large_p14_336_e1_gpu1_pretrain.py --deepspeed deepspeed_zero2
```
#### Remember to change the batch size and gradient accumulation parameters to fit your hardware. So your GPU_num * batch_size * gradient_accumulation is roughly equal to mine to reproduce the result.
The checkpoint and tensorboard logs are saved by default in ./work_dirs/. I only train it for 1 epoch to be same as the original LLaVA paper. Some researches also report that training for multiple epochs will make the model overfit the training dataset and perform worse in other domains.
This is my loss curve for llava-clip-internlm2-1_8b-pretrain-v1:
And the learning rate curve:
2. Instruction following fine-tuning
Please check the final release version |