Clinician-supervised large language AI model after-discharge instruction generation for common emergency department...

After-discharge instructions often fail due to poor usability and language misalignment. We evaluated a clinician-supervised method for generating instructions for common emergency department presentations using a clinician-supervised method using large language models.

Eight common ED presentations were identified via a physician needs assessment. After-discharge instructions were generated using three publicly accessible large language models (ChatGPT‑4.0, Claude‑3.5 Haiku, and Gemini‑2.0 Flash Thinking) and iteratively refined by expert physicians. After-discharge instructions were assessed for clinical accuracy, completeness, readability (Flesch Reading Ease scale), semantic similarity, and understandability. This analysis was repeated after the inclusion of reviewer edits. Five AI-simulated personas based on local marginalised patient profiles were used to identify comprehension barriers. We applied Bag‑of‑Words and BioClinical BERT similarity metrics to objectively quantify the semantic and contextual consistency of LLM outputs beyond what readability scores alone capture.

All large language models produced clinically accurate instructions. Physician edits improved accuracy but paradoxically reduced objective readability scores. Whilst Claude was preferred for simpler language after revisions, persona reviews revealed persistent medical jargon and vague instructions that could hinder understanding for marginalised groups.

Large language models with expert clinician supervision can create clinically accurate after-discharge instructions. However, clinician-led refinements decreased readability, increasing the risk of poorer post-visit patient understanding. AI-simulated personas may offer a scalable method to surface potential comprehension barriers in patient instructions, but must be followed by validation with real patients.