bibtext: @inproceedings{
  zhang2024forcing,
  title={Forcing Diffuse Distributions out of Language Models},
  author={Yiming Zhang and Avi Schwarzschild and Nicholas Carlini and J Zico Kolter and Daphne Ippolito},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=9JY1QLVFPZ}
  }
abstract: Despite being trained specifically to follow user instructions, today’s instruction-tuned
  language models perform poorly when instructed to produce random outputs.
  For example, when prompted to pick a number uniformly between one and
  ten Llama-2-13B-chat disproportionately favors the number five, and when tasked
  with picking a first name at random, Mistral-7B-Instruct chooses Avery 40 times
  more often than we would expect based on the U.S. population. When these language
  models are used for real-world tasks where diversity of outputs is crucial,
  such as language model assisted dataset construction, their inability to produce
  diffuse distributions over valid choices is a major hurdle. In this work, we propose
  a fine-tuning method that encourages language models to output distributions that
  are diffuse over valid outcomes. The methods we introduce generalize across a
  variety of tasks and distributions and make large language models practical for
  synthetic dataset generation with little human intervention.
title: Forcing Diffuse Distributions out of Language Models
authors: Yiming Zhang Avi Schwarzschild Nicholas Carlini J Zico Kolter Daphne Ippolito