arxiv:2501.13007

Pairwise RM: Perform Best-of-N Sampling with Knockout Tournament

Published on Jan 22

· Submitted by

RicardoL1u on Jan 23

Upvote

Authors:

Juanzi Li

Abstract

Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large Language Models (LLMs), relies on reward models to select the best candidate solution from multiple generations. However, traditional reward models often assign arbitrary and inconsistent scores, limiting their effectiveness. To address this, we propose a Pairwise Reward Model (Pairwise RM) combined with a knockout tournament for BoN sampling. Instead of assigning absolute scores, given one math problem, Pairwise RM evaluates two candidate solutions' correctness simultaneously. This approach eliminates the need for arbitrary scoring and enables cross-validation of solutions through parallel comparison. In the knockout tournament, Pairwise RM conducts pairwise comparisons between candidate solutions and eliminates the incorrect ones iteratively. We construct \ourdataset, a large-scale dataset of 443K pairwise comparisons derived from NumiaMath and annotated using gemini-1.5-flash, and train the Pairwise RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench demonstrate significant improvements over traditional discriminative reward models. And a 40\% to 60\% relative improvement is achieved on the top 50\% challenging problems.

View arXiv page View PDF Add to collection

Community

RicardoL1u

Paper submitter about 15 hours ago

•

edited about 15 hours ago

🤓 Introducing Pairwise RM: A Better Way to Rank AI Solutions! ✨

In reasoning tasks, Best-of-N (BoN) sampling is a common method where a Large Language Model (LLM) generates multiple candidate solutions for a given problem, and the best one is selected using a reward model.
However, traditional reward models achieve this by assigning absolute scores to each solution, which are found to be unstable and inconsistent. 🤔

To address this, we propose Pairwise Reward Model (Pairwise RM), a new approach combined with a knockout tournament for more stable and reliable ranking of AI solutions. 🏆

Here's how it works:

1️⃣ Side-by-Side Comparisons: Unlike traditional reward models, Pairwise RM evaluates two solutions simultaneously to veirfy correctness. This avoids the inconsistency of absolute numerical scoring and enables a cross-validation mechanism. 🔄

2️⃣ Knockout Tournament: Candidate solutions go through iterative rounds of pairwise elimination, ensuring that only the best solution remains. 🥇

To build Pairwise RM, we created PAIRWISE-443K, a large-scale dataset of pairwise comparisons for math solutions. 📊

Key Results:

Better Performance: Our experiments show that Pairwise RM outperforms traditional reward models in both MATH-500 and Olympiad Bench. 📈
Challenging Prolbems: On the top 50% of challenging problems, Pairwise RM achieved a 40% to 60% relative improvement compared to traditional methods. 🚀

Check out our work for more details and see how Pairwise RM could improve BoN sampling! 📚 GitHub Link 🔗
Arxiv Link 🔗