bibtext: @inproceedings{
  li2024measuring,
  title={Measuring and Controlling Instruction (In)Stability in Language Model Dialogs},
  author={Kenneth Li and Tianle Liu and Naomi Bashkansky and David Bau and Fernanda Vi{\'e}gas and Hanspeter Pfister and Martin Wattenberg},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=60a1SAtH4e}
  }
abstract: System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be _stable_, so the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, we reveal a significant _instruction drift_ within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to _attention decay_ over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called split-softmax, which compares favorably against two strong baselines.  
  Code: [https://github.com/likenneth/persona_drift](https://github.com/likenneth/persona_drift).
title: Measuring and Controlling Instruction (In)Stability in Language Model Dialogs
authors: Kenneth Li Tianle Liu Naomi Bashkansky David Bau Fernanda Viégas Hanspeter Pfister Martin Wattenberg