bibtext: @inproceedings{
  zhu2024starlingb,
  title={Starling-7B: Improving Helpfulness and Harmlessness with {RLAIF}},
  author={Banghua Zhu and Evan Frick and Tianhao Wu and Hanlin Zhu and Karthik Ganesan and Wei-Lin Chiang and Jian Zhang and Jiantao Jiao},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=GqDntYTTbk}
  }
abstract: This paper presents Starling-7B, the current best-performing 7B chat model on Chatbot Arena, along with its training dataset Nectar, a high-quality preference dataset collected by prompting GPT-4 to rank responses. We propose an internal pairwise rating technique, where the model considers all pairings before providing a ranking decision, leveraging the proven pairwise rating capability of LLMs without the cost of individual pairwise calls. The resulting Nectar dataset comprises 182,954 chat prompts, each with seven responses from various models, ranked by GPT-4, equating to 3.8 million high-quality pairwise comparisons. We introduce Starling-RM-7B and Starling-RM-34B, the reward model suites trained with a K-wise preference loss on Nectar, outperforming pairwise counterparts. We benchmark reward model training pipelines across metrics such as human preference, truthfulness, and safety. Using Nectar and our new training pipeline, we fine-tuned Openchat-3.5 to create Starling-LM-7B, achieving significant performance enhancements on MT-Bench, AlpacaEval, and human evaluation metrics. To facilitate research and understanding of RLHF mechanisms, we open-source the Nectar dataset, the reward models, and the language models.
title: Starling-7B: Improving Helpfulness and Harmlessness with RLAIF
authors: Banghua Zhu Evan Frick Tianhao Wu Hanlin Zhu Karthik Ganesan Wei-Lin Chiang Jian Zhang Jiantao Jiao