bibtext: @inproceedings{
  styles2024workbench,
  title={WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting},
  author={Olly Styles and Sam Miller and Patricio Cerda-Mardini and Tanaya Guha and Victor Sanchez and Bertie Vidgen},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=4HNAwZFDcH}
  }
abstract: We introduce WorkBench: a benchmark dataset for evaluating agents’ ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents’ errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents’ ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/link_updated_upon_acceptance
title: WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
authors: Olly Styles Sam Miller Patricio Cerda-Mardini Tanaya Guha Victor Sanchez Bertie Vidgen