Large language model-based scribing tools for ophthalmology: performance and safety evaluation using simulated...

To compare the performance and safety of six LLM-based scribing tools in generating ophthalmological clinical notes from simulated physician-patient encounters.

This cross-sectional comparative study involved seven ophthalmic encounters, transcribed by six scribing tools (Heidi AI, Whisper GPT-4o, Whisper Gemini 1.5 Pro, i-scribe, Lyrebird, Twofold). Scribing tools were prompted to generate notes using Subjective, Objective, Assessment, and Plan (SOAP) format. Consultant-level ophthalmologists evaluated note quality using the Physician Documentation Quality Instrument (PDQI-9), which evaluates eight dimensions: "accurate", "thorough", "useful", "organized", "comprehensive", "succinct", "synthesized", "internally consistent" on a 5-point scale (with a maximum sum of 40). Text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore) were also used. Safety analysis was conducted to identify documentation errors categorised as omissions, incorrect information, and extraneous content.

Heidi AI achieved the highest overall documentation quality (PDQI-9: 38.524, 95%CI: 37.815-39.233), followed by Whisper GPT-4o (38.048, 95%CI: 36.747-39.349). It scored highest in accuracy, usefulness, organisation, comprehensiveness, and succinctness, and led in ROUGE-L (0.326, 95%CI: 0.276-0.374), BERTScore (0.727, 95%CI: 0.707-0.747), and AlignScore (0.704, 95%CI: 0.650-0.758). All tools, however, exhibited errors. The most common were omissions of clinically relevant details, including disease severity, coexisting ocular conditions, systemic comorbidities, and management specifics. Some tools also generated incorrect or extraneous content, such as hallucinated patient requests or incorrect patient names.

Among the evaluated LLM based scribing tools, Heidi AI demonstrated the highest performance and shows promise for supporting ophthalmology documentation. However, persistent errors, particularly omissions of critical details, were observed across all tools and could compromise patient care. Human oversight and rigorous verification processes remain essential before clinical integration.