Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
arxiv(2023)
摘要
Previous works in prompt engineering for large language models have
introduced different gradient-free probability-based prompt selection methods
that aim to choose the optimal prompt among the candidates for a given task but
have failed to provide a comprehensive and fair comparison between each other.
In this paper, we propose a unified framework to interpret and evaluate the
existing probability-based prompt selection methods by performing extensive
experiments on 13 common and diverse NLP tasks. We find that each of the
existing methods can be interpreted as some variant of the method that
maximizes mutual information between the input and the predicted output (MI).
Utilizing this finding, we develop several other combinatorial variants of MI
and increase the effectiveness of the oracle prompt selection method from
87.79
prompt to that of the optimal oracle prompt. Furthermore, considering that all
the methods rely on the output probability distribution of the model that might
be biased, we propose a novel calibration method called Calibration by
Marginalization (CBM) that is orthogonal to the existing methods and helps
increase the prompt selection effectiveness of the best method to 96.85
achieving 99.44
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要