Evaluating Hospital Course Summarization by an Electronic Health Record-Based Large Language Model.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Hospital course (HC) summarization is a time-consuming task for physicians.
• Large language models (LLMs) show promise for automating HC summarization.
• The effectiveness of physician-LLM partnerships in electronic health record (EHR)-embedded workflows for drafting HCs is not well understood.

Purpose of the Study:

• To compare the editing effort required by resident physicians to refine LLM-generated HCs versus physician-generated HCs.
• To evaluate the quality of HCs based on the 4Cs criteria: completeness, conciseness, cohesiveness, and confabulation-free.

Main Methods:

• A quality improvement study involving 10 internal medicine residents and 8 hospitalists.
• Residents edited physician-generated and LLM-generated HCs for randomly selected admissions.
• Editing effort was measured by the percentage of edits and semantic change, with comparative ratings on the 4Cs using A/B testing.

Main Results:

• LLM-generated HCs required significantly less editing (31.5% vs. 44.8%) and semantic change (2.4% vs. 4.9%) compared to physician-generated HCs.
• Hospitalists rated LLM HCs as more complete (P<.001) and similarly concise (P=.20) and cohesive (P=.60).
• LLM HCs had more confabulations (P=.002), but overall composite scores were similar (P=.46).

Conclusions:

• EHR-embedded LLMs can assist physicians in drafting HCs, requiring less editing to meet quality standards.
• Physician-LLM partnerships show feasibility for improving HC generation efficiency and quality.
• Further monitoring of LLM-generated HCs in clinical practice is warranted, particularly regarding confabulations.