Improving Large Language Models' Summarization Accuracy by Adding Highlights to Discharge Notes: Comparative...

Area of Science:

• Natural Language Processing
• Medical Informatics
• Artificial Intelligence

Background:

• The American Medical Association recommends simplifying electronic health record (EHR) notes for patient readability.
• Large language models (LLMs) show promise in summarizing EHR notes but can introduce inaccuracies.
• This study focuses on the initial step of simplifying discharge notes using LLMs.

Purpose of the Study:

• To test if LLM-generated summaries of highlighted discharge notes are more accurate than summaries of original notes.
• To evaluate the impact of highlighting detailed information on summary quality.

Main Methods:

• 15 discharge notes from MIMIC III were sampled and detailed information was highlighted using a machine learning-developed interface terminology.
• GPT-4o was used with prompt engineering to generate summaries from both highlighted and unhighlighted notes.
• Summaries were manually evaluated for completeness, correctness, and structural integrity.

Main Results:

• Summaries from highlighted notes (H-summaries) achieved 96% completeness, 8% higher than unhighlighted notes (U-summaries) (P=.01).
• H-summaries demonstrated better correctness with fewer errors (2 vs 3) and less misplaced information (2 vs 8).
• Statistical significance was observed for completeness (P=.01) and header accuracy (P=.03).

Conclusions:

• Using highlighted discharge notes with LLMs and prompt engineering enhances summary quality.
• This approach improves correctness, completeness, and structural integrity compared to using unhighlighted notes.
• Highlighting detailed information is a key step in generating accurate and simplified EHR summaries.