bibtext: @inproceedings{
  wu2024hummer,
  title={Hummer: Towards Limited Competitive Preference Dataset},
  author={Yusen Wu and Li Jiang and Junwu Xiong and Jingqing Ruan and Yichuan Ding and Qingpei Guo and zujie wen and JUN ZHOU and Xiaotie Deng},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=aKwQPRjdGa}
  }
abstract: Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present \texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative pairwise preference datasets with reduced-conflict alignment objectives. \texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, \texttt{HummerRM} and \texttt{HummerRM-F}, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions \texttt{HummerRM} as an ideal model for domain-specific further fine-tuning and reducing vulnerability to jailbreak attacks.
title: Hummer: Towards Limited Competitive Preference Dataset
authors: Yusen Wu Li Jiang Junwu Xiong Jingqing Ruan Yichuan Ding Qingpei Guo zujie wen JUN ZHOU Xiaotie Deng