LQER: Low-Rank Quantization Error Reconstruction for LLMs
CoRR(2024)
摘要
Post-training quantization of Large Language Models (LLMs) is challenging. In
this work, we introduce Low-rank Quantization Error Reduction (LQER), which
combines quantization and low-rank approximation to recover the model
capability. LQER leverages an activation-induced scale matrix to drive the
singular value distribution of quantization error towards a desirable
distribution, which enables nearly-lossless W4A8 quantization on various LLMs
and downstream tasks without the need for knowledge distillation, grid search,
or gradient-base iterative optimization. Unlike existing methods, the
computation pattern of LQER eliminates the need for specialized Scatter and
Gather processes to collect high-precision weights from irregular memory
locations. Our W4A8 LLMs achieve near-lossless performance on six popular
downstream tasks, while using 1.36× fewer hardware resources than the
leading state-of-the-art method. We will open-source our framework once the
paper is accepted.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要