TP-Aware Dequantization
arxiv(2024)
摘要
In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要