bibtext: @inproceedings{
  fan2024polyvisualexpert,
  title={Poly-Visual-Expert Vision-Language Models},
  author={Xiaoran Fan and Tao Ji and 江常皓 and Shuo Li and Senjie Jin and Sirui Song and Junke Wang and Boyang Hong and Lu Chen and Guodong Zheng and Ming Zhang and Huangcaishuang and Rui Zheng and Zhiheng Xi and Yuhao Zhou and Shihan Dou and Junjie Ye and Hang Yan and Tao Gui and Qi Zhang and Xipeng Qiu and Xuanjing Huang and Zuxuan Wu and Yu-Gang Jiang},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=7QaEO9WYMa}
  }
abstract: Current large vision-language models (VLMs) frequently face challenges such as the limited capabilities of a single visual component and the excessive length of visual tokens. These issues can limit the model's ability to interpret complex visual information and over-lengthy contextual information accurately. Tackling these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes leveraging the ensemble experts technique to synergize the capabilities of individual visual encoders, including those skilled in image-text matching, image segmentation, OCR, etc. This method introduces a fusion network that consolidates the outputs from different visual experts while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to mitigate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient 64 or even down to 1. Experimental results show that VLMs with multiple experts consistently outperform isolated visual encoders, with notable performance improvements as more experts are integrated. Our codes are available on our project website.
title: Poly-Visual-Expert Vision-Language Models
authors: Xiaoran Fan Tao Ji 江常皓 Shuo Li Senjie Jin Sirui Song Junke Wang Boyang Hong Lu Chen Guodong Zheng Ming Zhang Huangcaishuang Rui Zheng Zhiheng Xi Yuhao Zhou Shihan Dou Junjie Ye Hang Yan Tao Gui Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yu-Gang Jiang