Physicians Outperform Large Language Models in Pediatric Discharge Summary Generation.

Area of Science:

• Medical Informatics
• Clinical Documentation
• Artificial Intelligence in Healthcare

Background:

• Pediatric hospitalists face increasing patient complexity and workload.
• Large language models (LLMs) show potential for automating clinical documentation, including discharge summaries.
• Assessing the quality of LLM-generated clinical documentation is crucial for evaluating their utility.

Purpose of the Study:

• To compare the quality of unedited LLM-generated discharge summaries with physician-authored discharge summaries.
• To evaluate clinician perceptions of LLM-generated versus physician-authored pediatric discharge summaries.

Main Methods:

• A comparative evaluation of 35 unedited LLM-generated and 35 physician-authored discharge summaries.
• Summaries were anonymously graded by pediatric hospitalists and primary care pediatricians using the Physician Documentation Quality Instrument (PDQI)-9.
• Clinical Risk Group (CRG) and length of stay were collected for each summary.

Main Results:

• Physician-authored summaries received higher overall quality scores from both pediatric hospitalists and primary care pediatricians.
• LLM-generated summaries were significantly longer than physician-authored summaries.
• Higher CRG scores correlated with decreased quality scores for physician-authored summaries.

Conclusions:

• Physicians produced higher-quality discharge summaries compared to unedited LLMs.
• Future research should investigate physician-modified LLM documentation for potential improvements in quality and efficiency.
• The impact of LLMs on physician workload and well-being warrants further study.