bibtext: @inproceedings{
  shnitzer2024large,
  title={Large Language Model Routing with Benchmark Datasets},
  author={Tal Shnitzer and Anthony Ou and M{\'\i}rian Silva and Kate Soule and Yuekai Sun and Justin Solomon and Neil Thompson and Mikhail Yurochkin},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=Zb0ajZ7vAt}
  }
abstract: The number of open-source Large Language Models (LLMs) grows daily, as does the number of available benchmark datasets used to evaluate LLMs. While some models dominate these benchmarks, no single model achieves the best accuracy in all tasks and use cases. In light of this observation, we address the challenge of selecting the best LLM from a collection of pre-trained models, given a new task.  While related work relies on evaluating each candidate model on a set of labeled examples, our new formulation does not assume any labeled data from the new task is available.  Instead, we repurpose a collection of benchmark datasets---which may focus on different tasks than the one at hand---to learn a ''router'' model for  LLM selection from inputs only; this problem reduces to a collection of binary classification tasks. Empirically, our strategy consistently improves performance over using any single model for all tasks.
title: Large Language Model Routing with Benchmark Datasets
authors: Tal Shnitzer Anthony Ou Mírian Silva Kate Soule Yuekai Sun Justin Solomon Neil Thompson Mikhail Yurochkin