Clinician-supervised large language AI model after-discharge instruction generation for common emergency department...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Patient Communication

Background:

• After-discharge instructions are crucial for patient recovery but often suffer from poor usability and language barriers.
• Effective patient communication post-discharge is essential to prevent readmissions and improve health outcomes.

Purpose of the Study:

• To evaluate a clinician-supervised method for generating emergency department after-discharge instructions using large language models (LLMs).
• To assess the clinical accuracy, readability, and understandability of LLM-generated instructions, and the impact of physician refinement.

Main Methods:

• Eight common emergency department presentations were selected.
• Instructions were generated using ChatGPT-4.0, Claude-3.5 Haiku, and Gemini-2.0 Flash Thinking, then refined by physicians.
• Assessments included clinical accuracy, completeness, readability (Flesch Reading Ease), semantic similarity (Bag-of-Words, BioClinical BERT), and understandability using AI-simulated patient personas.

Main Results:

• All LLMs produced clinically accurate instructions.
• Physician edits improved accuracy but reduced objective readability scores.
• Claude-3.5 Haiku showed preference for simpler language post-revision, yet persona reviews identified persistent jargon and vagueness impacting marginalized groups.

Conclusions:

• Clinician-supervised LLMs can produce clinically accurate after-discharge instructions.
• Physician refinement paradoxically decreased readability, potentially worsening patient comprehension.
• AI-simulated personas can identify comprehension barriers, but real-patient validation is necessary.