bibtext: @inproceedings{
  ou2024adaptive,
  title={Adaptive Quantization Error Reconstruction for {LLM}s with Mixed Precision},
  author={Lin Ou and Jinpeng Xia and Yuewei Zhang and Chuzhan Hao and Hao Henry Wang},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=T5pGDydMkS}
  }
abstract: Large language models (LLMs) has demonstrated superior performance on various downstream tasks. However, their practical applications are hindered by their immense memory and computation requirements. Although recent post-training quantization methods can effectively reduce memory usage and improve computational efficiency, they often overlook the varying sensitivity of different layer weights to bit precision. Additionally, the previous methods suffer from significant accuracy loss under low-bit quantization (2-3 bits). To address these limitations, we propose Adaptive Mixed Precision and Low-Rank Quantization Error Reconstruction for LLMs (AMLQ), which achieves state-of-the-art performance under the approximate average bit precision overall. Furthermore, we introduce the low-rank decomposition to reconstruct quantization error based on the output features. Experimental results demonstrate that this method can be effectively combined with various quantization techniques and bring considerable performance gains. Our approach comprehensively considers model performance and inference efficiency, offering more than 3$\times$ speedup over the FP16 execution.
title: Adaptive Quantization Error Reconstruction for LLMs with Mixed Precision
authors: Lin Ou Jinpeng Xia Yuewei Zhang Chuzhan Hao Hao Henry Wang