bibtext: @inproceedings{
  rein2024gpqa,
  title={{GPQA}: A Graduate-Level Google-Proof Q\&A Benchmark},
  author={David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=Ti67584b98}
  }
abstract: We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
  We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65\% accuracy (74\% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34\% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are "Google-proof"). When we released this dataset in November 2023, GPT-4 achieved 39\% accuracy. As of March 2024, Claude 3 Opus achieves a reported score of approximately 60\%, highlighting the rapid pace of progress in AI. If we are to use future AI systems to help us answer very hard questions—for example, when developing new scientific knowledge—we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA for skilled non-experts should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.
title: GPQA: A Graduate-Level Google-Proof Q   A Benchmark
authors: David Rein Betty Li Hou Asa Cooper Stickland Jackson Petty Richard Yuanzhe Pang Julien Dirani Julian Michael Samuel R. Bowman