Factcheck-GPT: End-to-End Fine-Grained Document-Level Fact-Checking and Correction of LLM Output.

Yuxia Wang,Revanth Gangi Reddy, Zain Muhammad Mujahid,Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal,Liangming Pan,Nadav Borenstein, Aditya Pillai,Isabelle Augenstein,Iryna Gurevych,Preslav Nakov

CoRR(2023)

引用 0|浏览6
暂无评分
摘要
The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We design and build an annotation tool to speed up the labelling procedure and ease the workload of raters. It allows flexible incorporation of automatic results in any stage, e.g. automatically-retrieved evidence. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims with the best F1=0.53. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.
更多
查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络