bibtext: @inproceedings{
  tunstall2024zephyr,
  title={Zephyr: Direct Distillation of {LM} Alignment},
  author={Lewis Tunstall and Edward Emanuel Beeching and Nathan Lambert and Nazneen Rajani and Kashif Rasul and Younes Belkada and Shengyi Huang and Leandro Von Werra and Cl{\'e}mentine Fourrier and Nathan Habib and Nathan Sarrazin and Omar Sanseviero and Alexander M Rush and Thomas Wolf},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=aKkAwZB6JV}
  }
abstract: We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, set a new state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpassed Llama2-Chat-70B, at the time the best open-access RLHF-based model.
title: Zephyr: Direct Distillation of LM Alignment
authors: Lewis Tunstall Edward Emanuel Beeching Nathan Lambert Nazneen Rajani Kashif Rasul Younes Belkada Shengyi Huang Leandro Von Werra Clémentine Fourrier Nathan Habib Nathan Sarrazin Omar Sanseviero Alexander M Rush Thomas Wolf