bibtext: @inproceedings{
  zhang2024ferretv,
  title={Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models},
  author={Haotian Zhang and Haoxuan You and Philipp Dufter and Bowen Zhang and Chen Chen and Hong-You Chen and Tsu-Jui Fu and William Yang Wang and Shih-Fu Chang and Zhe Gan and Yinfei Yang},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=EEPBOB2Xww}
  }
abstract: While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.
title: Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
authors: Haotian Zhang Haoxuan You Philipp Dufter Bowen Zhang Chen Chen Hong-You Chen Tsu-Jui Fu William Yang Wang Shih-Fu Chang Zhe Gan Yinfei Yang