Developing and Evaluating Large Language Model-Generated Emergency Medicine Handoff Notes

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Physician documentation burden in emergency medicine (EM) is significant.
• Handoffs from EM to inpatient (IP) settings require accurate and safe information transfer.
• Large language models (LLMs) offer potential for automating clinical documentation.

Purpose of the Study:

• To develop and evaluate LLM-generated EM-to-IP handoff notes.
• To compare the accuracy and patient safety of LLM-generated notes against physician-written notes.

Main Methods:

• A cohort study of 1600 EM patient records from NewYork-Presbyterian/Weill Cornell Medical Center (2023).
• A customized clinical LLM pipeline was developed to generate templated EM-to-IP handoff notes.
• Evaluation used automated metrics (ROUGE, BERTScore, SCALE) and a novel patient safety framework, including physician review.

Main Results:

• LLM-generated notes demonstrated higher scores in lexical similarity (ROUGE, BERTScore) and fidelity (SCALE) compared to physician-written notes.
• Physician review of a subsample indicated LLM notes were marginally less useful (4.04/5 vs 4.36/5) and safe (4.06/5 vs 4.50/5).
• No critical patient safety risks were identified in the LLM-generated summaries.

Conclusions:

• LLM-generated EM-to-IP handoff notes show promise, outperforming physician notes on automated metrics.
• A slight inferiority in usefulness and safety suggests the need for a physician-in-the-loop approach for LLM implementation.
• The study provides a framework for measuring the preimplementation patient safety of LLM clinical tools.