A Benchmark of Domain-Adapted Large Language Models for Generating Brief Hospital Course Summaries
arxiv(2024)
摘要
Brief hospital course (BHC) summaries are common clinical documents generated
by summarizing clinical notes. While large language models (LLMs) depict
remarkable capabilities in automating real-world tasks, their capabilities for
healthcare applications such as BHC synthesis have not been shown. To enable
the adaptation of LLMs for BHC synthesis, we introduce a novel benchmark
consisting of a pre-processed dataset extracted from MIMIC-IV notes,
encapsulating clinical note, and brief hospital course (BHC) pairs. We assess
the performance of two general-purpose LLMs and three healthcare-adapted LLMs
to improve BHC synthesis from clinical notes. Using clinical notes as input for
generating BHCs, we apply prompting-based (using in-context learning) and
fine-tuning-based adaptation strategies to three open-source LLMs
(Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5,
GPT-4). We quantitatively evaluate the performance of these LLMs across varying
context-length inputs using conventional natural language similarity metrics.
We further perform a qualitative study where five diverse clinicians blindly
compare clinician-written BHCs and two LLM-generated BHCs for 30 samples across
metrics of comprehensiveness, conciseness, factual correctness, and fluency.
Overall, we present a new benchmark and pre-processed dataset for using LLMs in
BHC synthesis from clinical notes. We observe high-quality summarization
performance for both in-context proprietary and fine-tuned open-source LLMs
using both quantitative metrics and a qualitative clinical reader study. We
propose our work as a benchmark to motivate future works to adapt and assess
the performance of LLMs in BHC synthesis.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要