bibtext: @inproceedings{
  ko2024on,
  title={On Robustness-Accuracy Characterization of Language Models using Synthetic Datasets},
  author={Ching-Yun Ko and Pin-Yu Chen and Payel Das and Yung-Sung Chuang and Luca Daniel},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=C0j44uRPcl}
  }
abstract: In recent years, language models (LMs) that were pretrained at scale on diverse data have proven to be a successful approach for solving different downstream tasks. However, new concerns about proper performance evaluation have been raised, especially for test-data leakage caused by accidentally including them during pretraining, or by indirectly exposing them through API calls for evaluation. Motivated by these, in this paper, we propose a new evaluation workflow that generates steerable synthetic language datasets and proxy tasks for benchmarking the performance of pre-trained LMs on sentence classification tasks. This approach allows for better characterization of the joint analysis on the robustness and accuracy of LMs without risking sensitive information leakage. It also provides a more controlled and private way to evaluate LMs that avoids overfitting specific test sets. Verified on various pretrained LMs, the proposed approach demonstrates promising high correlation with real downstream performance.
title: On Robustness-Accuracy Characterization of Language Models using Synthetic Datasets
authors: Ching-Yun Ko Pin-Yu Chen Payel Das Yung-Sung Chuang Luca Daniel