bibtext: @inproceedings{
  duanmu2024skvq,
  title={{SKVQ}: Sliding-window Key and Value Cache Quantization for Large Language Models},
  author={Haojie Duanmu and Zhihang Yuan and Xiuhong Li and Jiangfei Duan and Xingcheng ZHANG and Dahua Lin},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=nI6JyFSnyV}
  }
abstract: Large language models (LLMs) have demonstrated the capability to process extended token sequences, enabling complex tasks such as book comprehension and long-form text generation. However, as context length increases, the key-value (KV) cache required for LLMs consumes substantial memory, becoming a bottleneck for deployment.
  This paper introduces SKVQ (Sliding-window KV cache Quantization), a strategy designed to address the challenge of extremely low bitwidth KV cache quantization. SKVQ rearranges the channels of the KV cache to enhance channel similarity within quantization groups and applies clipped dynamic quantization at the group level. Furthermore, SKVQ maintains high precision for the most recent window tokens in the KV cache, preserving accuracy for a small yet critical portion of the cache.
  Our evaluation of LLMs demonstrates that SKVQ achieves high compression ratios while maintaining accuracy, outperforming previous quantization methods. SKVQ enables the quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal accuracy loss. This advancement allows processing context lengths of up to 1M tokens on an 80GB GPU for a 7B parameter model, resulting in up to 7 times faster decoding.
title: SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
authors: Haojie Duanmu Zhihang Yuan Xiuhong Li Jiangfei Duan Xingcheng ZHANG Dahua Lin