bibtext: @inproceedings{
  liu2024visualwebbench,
  title={VisualWebBench: How Far Have Multimodal {LLM}s Evolved in Web Page Understanding and Grounding?},
  author={Junpeng Liu and Yifan Song and Bill Yuchen Lin and Wai Lam and Graham Neubig and Yuanzhi Li and Xiang Yue},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=egVSgtJJAx}
  }
abstract: Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce VisualWebBench, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. VisualWebBench consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites,  covering 87 sub-domains. We evaluate 16 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on VisualWebBench, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe VisualWebBench will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.
title: VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
authors: Junpeng Liu Yifan Song Bill Yuchen Lin Wai Lam Graham Neubig Yuanzhi Li Xiang Yue