Ranking Large Language Models without Ground Truth
CoRR(2024)
摘要
Evaluation and ranking of large language models (LLMs) has become an
important problem with the proliferation of these models and their impact.
Evaluation methods either require human responses which are expensive to
acquire or use pairs of LLMs to evaluate each other which can be unreliable. In
this paper, we provide a novel perspective where, given a dataset of prompts
(viz. questions, instructions, etc.) and a set of LLMs, we rank them without
access to any ground truth or reference responses. Inspired by real life where
both an expert and a knowledgeable person can identify a novice our main idea
is to consider triplets of models, where each one of them evaluates the other
two, correctly identifying the worst model in the triplet with high
probability. We also analyze our idea and provide sufficient conditions for it
to succeed. Applying this idea repeatedly, we propose two methods to rank LLMs.
In experiments on different generative tasks (summarization, multiple-choice,
and dialog), our methods reliably recover close to true rankings without
reference data. This points to a viable low-resource mechanism for practical
use.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要