Physician- and Large Language Model-Generated Hospital Discharge Summaries

Area of Science:

• Medical Informatics
• Clinical Documentation
• Artificial Intelligence in Healthcare

Background:

• High-quality discharge summaries are crucial for patient outcomes but increase physician documentation burden.
• Large language models (LLMs) offer a potential solution for drafting discharge summary narratives, aiding physicians.

Purpose of the Study:

• To evaluate the quality and safety of LLM-generated discharge summary narratives compared to physician-generated ones.
• To determine if LLM narratives are a viable tool for supporting clinical documentation.

Main Methods:

• A cross-sectional study of 100 inpatient hospital medicine encounters.
• Blinded, duplicate evaluation of physician- and LLM-generated narratives by 22 attending physicians.
• Assessment of overall quality, reviewer preference, comprehensiveness, concision, coherence, and error types (inaccuracies, omissions, hallucinations) with harmfulness scoring.

Main Results:

• LLM and physician narratives showed comparable overall quality (mean score 3.67 vs 3.77) and reviewer preference.
• LLM narratives were more concise and coherent but less comprehensive than physician narratives.
• LLM narratives had more unique errors (2.91 vs 1.82) but similar low overall potential for harm (mean score 0.84 vs 0.36).

Conclusions:

• LLM-generated discharge summary narratives are comparable in quality and equally preferred to physician-generated ones.
• Despite a higher error rate, LLM narratives demonstrate low overall harmfulness.
• LLM-generated narratives, with human oversight, represent a viable option for hospitalists to reduce documentation burden.