Artificial Intelligence-Generated Answers to Patients' Questions on Asthma: The Artificial Intelligence Responses on...

Asthma is a prevalent chronic respiratory disease requiring ongoing patient education and individualized management. The increasing reliance on digital tools, particularly generative artificial intelligence (AI), to answer health-related questions has raised concerns about the accuracy, reliability, and comprehensibility of AI-generated information for people living with asthma.

To evaluate systematically the reliability, accuracy, comprehensiveness, and understandability of responses generated by three widely used artificial intelligence-based chatbots (ChatGPT, Bard, and Copilot) to common questions formulated by people with asthma.

In this cross-sectional study, 15 questions regarding asthma management were formulated by patients and categorized by difficulty. Responses from ChatGPT, Bard, and Copilot were evaluated by international experts for accuracy and comprehensiveness, and by patient representatives for understandability. Reliability was assessed through consistency testing across devices. We conducted a blinded evaluation.

A total of 21 experts and 16 patient representatives participated in the evaluation. ChatGPT demonstrated the highest reliability (15 of 15 responses), accuracy (median score, 9.0; interquartile range [IQR], 7.0-9.0), and comprehensiveness (median score, 8.0; IQR, 8.0-9.0) compared with Bard and Copilot (P < .0001). Bard achieved superior scores in understandability (median score, 9.0; IQR, 8.0-10.0; P < .0001). Performance differences were consistent across question difficulty levels.

Artificial intelligence-driven chatbots can provide generally accurate and understandable responses to asthma-related questions. Variability in reliability and accuracy underscores the need for caution in clinical contexts. Artificial intelligence tools may complement but cannot replace professional medical advice in asthma management.