A dataset and benchmark for hospital course summarization with adapted large language models

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Brief hospital course (BHC) summaries are crucial clinical documents.
• Automating BHC synthesis from clinical notes using large language models (LLMs) is an emerging area.
• Existing LLM capabilities in healthcare summarization require further investigation.

Purpose of the Study:

• To introduce the MIMIC-IV-BHC dataset for adapting LLMs to BHC synthesis.
• To benchmark the summarization performance of general-purpose and healthcare-adapted LLMs.
• To evaluate LLM-generated BHCs for clinical decision-making enhancement.

Main Methods:

• Utilized clinical notes as input for LLMs.
• Applied prompting-based and fine-tuning-based adaptation strategies.
• Evaluated LLMs using quantitative metrics (BLEU, BERT-Score) and a qualitative clinical reader study with 5 clinicians.

Main Results:

• Fine-tuned Llama2-13B outperformed other domain-adapted models on quantitative metrics.
• GPT-4 with in-context learning showed robustness to increasing context lengths.
• Clinicians significantly preferred GPT-4 generated summaries over fine-tuned Llama2-13B and original summaries (P<.001).

Conclusions:

• The MIMIC-IV-BHC dataset and LLM performance benchmark are released.
• Both proprietary and open-source LLMs show high-quality summarization performance.
• LLM-generated summaries, particularly from GPT-4, show potential for improving clinical decision-making.