AI-generated patient information leaflets for oral anticoagulants: quality, usability, readability, and...

Oral anticoagulants are frequently associated with preventable hospital admissions, and patient education is essential to their safe and effective use. Patients increasingly use large language models (LLMs) for medication information, yet few studies have compared AI-generated outputs with regulatory materials or examined oral anticoagulants, a class in which misunderstanding can cause bleeding or thrombosis.

To assess the informational quality, usability, readability, and output reproducibility of AI-generated patient information leaflets (PILs) for oral anticoagulants compared with FDA-referenced patient materials.

PILs for five oral anticoagulants (warfarin, apixaban, dabigatran, rivaroxaban, and edoxaban) were generated by ChatGPT, Gemini, and DeepSeek using a standardised zero-shot prompt based on FDA labelling templates; FDA-approved leaflets served as the comparator. Materials were anonymised and brand-blinded. Three clinical pharmacists independently evaluated each PIL using the Patient Education Materials Assessment Tool for Print Materials (PEMAT-P) and a modified DISCERN (mDISCERN), which assess informational quality, understandability, and actionability, but not pharmacological accuracy or clinical safety. Readability was assessed using seven validated indices. Output reproducibility was examined within the same day and at Day 1, 14, and 28.

ChatGPT produced PILs of informational quality similar to FDA-referenced materials, both achieving a median mDISCERN score of (41/65); DeepSeek (34/65) and Gemini (33/65) scored significantly lower than ChatGPT; Gemini also scored significantly lower than the FDA-referenced materials. Understandability was acceptable for all sources, whereas actionability was limited, including in FDA materials, with no leaflet providing a patient summary or decision-support tool. All materials exceeded the recommended sixth- to eighth-grade range on most indices, with FDA leaflets the most complex. Output was stable across generations, with no significant within-model differences and the largest mean difference of 0.23 points.

Among the evaluated models, ChatGPT most closely matched FDA-referenced materials in the assessed informational domains and demonstrated stable output over 28 days. Shared deficiencies across AI-generated and FDA leaflets suggest broader limitations in written health information. This comparability was limited to the assessed informational domains and does not establish equivalence in pharmacological accuracy, clinical correctness, or patient safety. AI-generated PILs may serve as clinician-reviewed supplementary materials and should not be used as standalone tools without independent professional review and verification of clinical content.