bibtext: @inproceedings{
  dingjie2024milebench,
  title={MileBench: Benchmarking {MLLM}s in Long Context},
  author={Song Dingjie and Shunian Chen and Guiming Hardy Chen and Fei Yu and Xiang Wan and Benyou Wang},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=Uhwze2LEwq}
  }
abstract: Despite the rapid progression of Multimodal Large Language Models (MLLMs) and their impressive performance on various benchmarks, the applicability of these results to real-world tasks remains uncertain. 
  This ambiguity primarily stems from the benchmarks' limited consideration for long-context and multi-image tasks, which are critical elements in real-world applications.
  Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on time-series captioning tasks, potentially masking MLLMs' performance challenges such as hallucination in long-context situations. 
  To address these limitations, we introduce \textbf{\dataset}, a pioneering benchmark designed to rigorously test the \textbf{M}ult\textbf{I}modal \textbf{L}ong-cont\textbf{E}xt capabilities of MLLMs. 
  This benchmark comprises a mix of text and images, long contexts, multiple tasks, and tasks requiring both comprehension and generation. 
  We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' long-context adaptation capacity and their ability to complete tasks in long-context scenarios.
  Our experimental results, garnered from testing 19 models, revealed that while closed-source model GPT-4(Vision) outperforms others, most open-source MLLMs display inadequate performance in long-context situations. 
  Hence, we strongly encourage an intensification of research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.
title: MileBench: Benchmarking MLLMs in Long Context
authors: Song Dingjie Shunian Chen Guiming Hardy Chen Fei Yu Xiang Wan Benyou Wang