bibtext: @inproceedings{
  zhang2024data,
  title={Data Checklist: On Unit-Testing Datasets with Usable Information},
  author={Heidi Chenyu Zhang and Shabnam Behzad and Kawin Ethayarajh and Dan Jurafsky},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=XGJBEeziEb}
  }
abstract: Model checklists (Ribeiro et al., 2020) have emerged as a useful tool for understanding the behavior of LLMs, analogous to unit-testing in software engineering. However, despite datasets being a key determinant of model behavior, evaluating datasets -- e.g., for the existence of annotation artifacts -- is largely done ad hoc, once a problem in model behavior has already been found downstream.
  In this work, we take a more principled approach to unit-testing datasets by proposing a taxonomy based on the $\mathcal{V}$-information literature. We call a collection of such unit tests a data checklist.
  Using the checklist, not only are we able to recover known artifacts in well-known datasets such as SNLI, but we also discover previously unknown artifacts in preference datasets for LLM alignment.
  Data checklists further enable a new kind of data filtering, which we use to improve the efficacy and data efficiency of preference alignment.
title: Data Checklist: On Unit-Testing Datasets with Usable Information
authors: Heidi Chenyu Zhang Shabnam Behzad Kawin Ethayarajh Dan Jurafsky