AI-generated draft replies to patient messages: exploring effects of implementation.

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Electronic Health Records (EHRs) present administrative challenges for clinicians.
• Large Language Models (LLMs) offer potential solutions to reduce physician workload.
• Real-world validation of LLM tools in clinical settings is crucial for adoption.

Purpose of the Study:

• To evaluate the adoption, usage, and time-saving potential of LLM-generated draft responses to patient messages within an EHR system.
• To assess the impact of LLM integration on physician workflow and response quality.

Main Methods:

• A prospective observational cohort study involving 100 physicians across 14 specialties in a non-English academic hospital.
• Physicians chose between LLM-generated draft replies and blank replies for 919 patient messages over 16 weeks.
• Analysis included adoption rates, text similarity scores (ROUGE-1, BLEU-1), and time spent on message adjustments.

Main Results:

• Physicians adopted LLM-generated drafts for 58% of patient messages.
• 43% of adopted drafts showed significant text similarity (≥10% match, ROUGE-1: 86%) to the final sent message.
• No significant difference in total response time was observed between LLM-assisted and blank replies (153s vs. 157s).

Conclusions:

• General adoption of LLM-generated draft responses reached 58%, with significant variation across specialties.
• Current implementation did not yield immediate time savings, suggesting a learning curve.
• LLM integration shows promise for safe use in a tertiary, non-English clinical setting, with potential for future efficiency improvements.