Patient questions and artificial intelligence answers: Evaluating chatbots in aspirin-exacerbated respiratory disease

Background: Aspirin-exacerbated respiratory disease (AERD), also known as nonsteroidal anti-inflammatory drug-exacerbated respiratory disease, is a chronic condition that is both clinically complex and often difficult for patients to understand. As patients increasingly turn to online tools for medical information, it is important to evaluate the quality of the responses they may receive. Objective: This study assessed the medical accuracy and readability of responses generated by ChatGPT 5.1, Gemini 2.5 Flash, and Claude Sonnet 4.5 to 12 common questions that patients often ask their providers about AERD. Methods: The 12 questions were developed by clinicians and patients; the resulting chatbot responses were de-identified and reviewed by an expert panel of 11 AERD specialists who rated each response for medical accuracy on a 10-point scale. Readability was measured by using readability and grade-level analysis programs. Results: Gemini's responses had the highest overall ratings for medical accuracy, followed by Claude and ChatGPT. The models generated responses written at a postgraduate reading level, which is substantially higher than recommended targets for patient-education materials. Questions with more complex responses showed greater variability in expert ratings, potentially reflecting differences in clinical interpretation. Conclusion: Analysis of these findings suggests that, whereas chatbots may improve access to medical information, their current outputs require clinician review and substantial simplification before they can be reliably used for patient education.