Towards Inpatient Discharge Summary Automation via Large Language Models: A Multidimensional Evaluation with a...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show potential for automating clinical documentation, reducing clinician workload.
• Previous studies on LLMs for clinical tasks used older models and limited data, raising concerns about accuracy.
• Automating hospital discharge summaries is a key area for LLM application to improve efficiency.

Purpose of the Study:

• To evaluate the performance of OpenAI's GPT-4o for generating inpatient Internal Medicine discharge summaries.
• To compare AI-generated summaries against human-written summaries on key quality metrics.
• To assess the safety implications of AI-generated discharge summaries, including fabrications and omissions.

Main Methods:

• Utilized a HIPAA-compliant Microsoft Azure instance of OpenAI's GPT-4o.
• Generated inpatient Internal Medicine discharge summaries using the LLM.
• Internal Medicine hospital faculty scored both AI-generated and human-written summaries on quality, readability, factuality, completeness, and safety.
• Compared AI summaries to actual patient discharge summaries.

Main Results:

• AI-generated discharge summaries significantly outperformed human-written summaries in quality and readability/conciseness.
• AI summaries were comparable to human summaries in factuality and completeness.
• The cost associated with AI-generated summaries was minimal.
• Hallucinations and omissions were minimal and did not significantly impact safety.

Conclusions:

• GPT-4o demonstrates high potential for automating the generation of clinical discharge summaries.
• AI-generated summaries offer superior quality and readability compared to traditional human-written summaries.
• LLMs like GPT-4o can be safely and effectively integrated into clinical workflows for documentation tasks.