Evaluating the role of ChatGPT in patient questions regarding drug allergies

The use of artificial intelligence (AI) technology is increasing. It takes patients with drug allergies longer to get an appointment at allergy clinics. It remains unclear whether AI technology will be able to help patients with drug allergies. In this study, we aimed to evaluate ChatGPT-5's responses to frequently asked questions about drug allergies in terms of accuracy, currency, comprehensiveness, usefulness, and clarity of language.

Frequently asked real patient questions about drug allergies were reviewed by experienced allergy specialists, and the 10 most common and clinically relevant questions were submitted to ChatGPT-5. Forty independent allergy-immunology specialists reviewed these answers using a standardized rubric that evaluates accuracy, currency, comprehensiveness, usefulness, and clarity of language, each rated on a 1-5 Likert scale (1 = very poor, 5 = excellent).

No questions received a score of 1 on any item across all domains (accuracy, currency, comprehensiveness, usefulness, and understandability). The lowest overall index score was for accuracy (median 3.9, Q1-Q3: 3.6-4.3), while the highest was for understandability (median 4.4, Q1-Q3: 3.7-4.9). Overall index scores were compared across domains, and statistically significant differences were found between accuracy and the other domains (p < 0.05), with accuracy showing relatively lower median values compared to currency, comprehensiveness, usefulness, and understandability. The overall index percentages of responses scoring ≥4 were 71.5% for accuracy, 74.8% for currency, 77.8% for comprehensiveness, 76.5% for usefulness, and 79.0% for understandability.

ChatGPT-5 demonstrated high accuracy, currency, comprehensiveness, usefulness, and understandability. Given the higher prevalence of drug allergies and the growing patient demand for allergy immunology specialists, the support of AI models for providing information to patients about drug allergies in this regard is essential.