Thermometer: Towards Universal Calibration for Large Language Models
arxiv(2024)
摘要
We consider the issue of calibration in large language models (LLM). Recent
studies have found that common interventions such as instruction tuning often
result in poorly calibrated LLMs. Although calibration is well-explored in
traditional applications, calibrating LLMs is uniquely challenging. These
challenges stem as much from the severe computational requirements of LLMs as
from their versatility, which allows them to be applied to diverse tasks.
Addressing these challenges, we propose THERMOMETER, a calibration approach
tailored to LLMs. THERMOMETER learns an auxiliary model, given data from
multiple tasks, for calibrating a LLM. It is computationally efficient,
preserves the accuracy of the LLM, and produces better-calibrated responses for
new tasks. Extensive empirical evaluations across various benchmarks
demonstrate the effectiveness of the proposed method.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要