Characterizing Tradeoffs in Language Model Decoding with Informational Interpretations.
CoRR(2023)
摘要
We propose a theoretical framework for formulating language model decoder
algorithms with dynamic programming and information theory. With dynamic
programming, we lift the design of decoder algorithms from the logit space to
the action-state value function space, and show that the decoding algorithms
are consequences of optimizing the action-state value functions. Each component
in the action-state value function space has an information theoretical
interpretation. With the lifting and interpretation, it becomes evident what
the decoder algorithm is optimized for, and hence facilitating the arbitration
of the tradeoffs in sensibleness, diversity, and attribution.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要