Accelerating Production LLMs with Combined Token/Embedding Speculators
arxiv(2024)
摘要
This technical report describes the design and training of novel speculative
decoding draft models, for accelerating the inference speeds of large language
models in a production environment. By conditioning draft predictions on both
context vectors and sampled tokens, we can train our speculators to efficiently
predict high-quality n-grams, which the base model then accepts or rejects.
This allows us to effectively predict multiple tokens per inference forward
pass, accelerating wall-clock inference speeds of highly optimized base model
implementations by a factor of 2-3x. We explore these initial results and
describe next steps for further improvements.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要