From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsy Narratives
arxiv(2024)
摘要
In settings where most deaths occur outside the healthcare system, verbal
autopsies (VAs) are a common tool to monitor trends in causes of death (COD).
VAs are interviews with a surviving caregiver or relative that are used to
predict the decedent's COD. Turning VAs into actionable insights for
researchers and policymakers requires two steps (i) predicting likely COD using
the VA interview and (ii) performing inference with predicted CODs (e.g.
modeling the breakdown of causes by demographic factors using a sample of
deaths). In this paper, we develop a method for valid inference using outcomes
(in our case COD) predicted from free-form text using state-of-the-art NLP
techniques. This method, which we call multiPPI++, extends recent work in
"prediction-powered inference" to multinomial classification. We leverage a
suite of NLP techniques for COD prediction and, through empirical analysis of
VA data, demonstrate the effectiveness of our approach in handling
transportability issues. multiPPI++ recovers ground truth estimates, regardless
of which NLP model produced predictions and regardless of whether they were
produced by a more accurate predictor like GPT-4-32k or a less accurate
predictor like KNN. Our findings demonstrate the practical importance of
inference correction for public health decision-making and suggests that if
inference tasks are the end goal, having a small amount of contextually
relevant, high quality labeled data is essential regardless of the NLP
algorithm.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要