Update README.md
Browse files
README.md
CHANGED
@@ -11,9 +11,10 @@ pipeline_tag: image-text-to-text
|
|
11 |
|
12 |
![efficiency](./assets/singleGPU.png)
|
13 |
|
|
|
14 |
## 🌈 Update
|
15 |
|
16 |
-
* **[2024.09.05]** LongLLaVA repo is published!🎉
|
17 |
|
18 |
## Architecture
|
19 |
|
@@ -41,17 +42,52 @@ pipeline_tag: image-text-to-text
|
|
41 |
|
42 |
|
43 |
|
44 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
45 |
|
46 |
-
|
47 |
|
48 |
-
|
|
|
|
|
49 |
|
50 |
-
|
|
|
|
|
|
|
|
|
51 |
|
|
|
52 |
|
|
|
53 |
|
|
|
54 |
|
|
|
55 |
|
56 |
## Citation
|
57 |
|
@@ -65,4 +101,7 @@ pipeline_tag: image-text-to-text
|
|
65 |
primaryClass={cs.CL},
|
66 |
url={https://arxiv.org/abs/2409.02889},
|
67 |
}
|
68 |
-
```
|
|
|
|
|
|
|
|
11 |
|
12 |
![efficiency](./assets/singleGPU.png)
|
13 |
|
14 |
+
|
15 |
## 🌈 Update
|
16 |
|
17 |
+
* **[2024.09.05]** LongLLaVA repo is published!🎉 The Code will
|
18 |
|
19 |
## Architecture
|
20 |
|
|
|
42 |
|
43 |
|
44 |
|
45 |
+
## Results reproduction
|
46 |
+
|
47 |
+
|
48 |
+
### Evaluation
|
49 |
+
|
50 |
+
- Preparation
|
51 |
+
|
52 |
+
Get the model inference code from [Github](https://github.com/FreedomIntelligence/LongLLaVA).
|
53 |
+
|
54 |
+
```bash
|
55 |
+
git clone https://github.com/FreedomIntelligence/LongLLaVA.git
|
56 |
+
```
|
57 |
+
|
58 |
+
- Environment Setup
|
59 |
+
|
60 |
+
```bash
|
61 |
+
pip install -r requirements.txt
|
62 |
+
```
|
63 |
+
|
64 |
+
|
65 |
+
- Command Line Interface
|
66 |
+
|
67 |
+
```bash
|
68 |
+
python cli.py --model_dir path-to-longllava
|
69 |
+
```
|
70 |
+
|
71 |
|
72 |
+
- Model Inference
|
73 |
|
74 |
+
```python
|
75 |
+
query = 'What does the picture show?'
|
76 |
+
image_paths = ['image_path1'] # image or video path
|
77 |
|
78 |
+
from cli import Chatbot
|
79 |
+
bot = Chatbot(path-to-longllava)
|
80 |
+
output = bot.inference(query, image_paths)
|
81 |
+
print(output) # Prints the output of the model
|
82 |
+
```
|
83 |
|
84 |
+
## TO DO
|
85 |
|
86 |
+
- [ ] Release Data Construction Code
|
87 |
|
88 |
+
## Acknowledgement
|
89 |
|
90 |
+
- [LLaVA](https://github.com/haotian-liu/LLaVA): Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
|
91 |
|
92 |
## Citation
|
93 |
|
|
|
101 |
primaryClass={cs.CL},
|
102 |
url={https://arxiv.org/abs/2409.02889},
|
103 |
}
|
104 |
+
```
|
105 |
+
|
106 |
+
|
107 |
+
|