bibtext: @inproceedings{
  saxon2024benchmarks,
  title={Benchmarks as Microscopes: A Call for Model Metrology},
  author={Michael Saxon and Ari Holtzman and Peter West and William Yang Wang and Naomi Saphra},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=bttKwCZDkm}
  }
abstract: Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their models have generalized traits such as reasoning or open-domain language understanding based on these flawed metrics. The science and practice of LMs requires a new approach to benchmarking which measures  specific capabilities with dynamic assessments. To be confident in our metrics, we need a new discipline of *model metrology*---one which focuses on  how to generate benchmarks that predict performance under deployment. Motivated by our evaluation criteria, we outline how building a community of model metrology practitioners---one focused on building tools and studying how to measure system capabilities---is the best way to meet these needs to and add clarity to the AI discussion.
title: Benchmarks as Microscopes: A Call for Model Metrology
authors: Michael Saxon Ari Holtzman Peter West William Yang Wang Naomi Saphra