AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators
CoRR(2024)
摘要
With the rise of generative AI, automated fact-checking methods to combat
misinformation are becoming more and more important. However, factual claim
detection, the first step in a fact-checking pipeline, suffers from two key
issues that limit its scalability and generalizability: (1) inconsistency in
definitions of the task and what a claim is, and (2) the high cost of manual
annotation. To address (1), we review the definitions in related work and
propose a unifying definition of factual claims that focuses on verifiability.
To address (2), we introduce AFaCTA (Automatic Factual Claim deTection
Annotator), a novel framework that assists in the annotation of factual claims
with the help of large language models (LLMs). AFaCTA calibrates its annotation
confidence with consistency along three predefined reasoning paths. Extensive
evaluation and experiments in the domain of political speech reveal that AFaCTA
can efficiently assist experts in annotating factual claims and training
high-quality classifiers, and can work with or without expert supervision. Our
analyses also result in PoliClaim, a comprehensive claim detection dataset
spanning diverse political topics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要