ChatGPT Performance on 120 Interdisciplinary Allergology Questions-Systematic Evaluation With Clinical Error Impact...

Area of Science:

• Artificial Intelligence in Medicine
• Allergy and Immunology Research
• Clinical Decision Support Systems

Background:

• Patients increasingly use AI tools like ChatGPT for medical information, including allergy-related queries.
• Long wait times for allergology appointments drive patients to seek alternative information sources.
• ChatGPT, while accessible, may provide inaccurate or incomplete medical advice, posing risks.

Purpose of the Study:

• To systematically evaluate the performance of ChatGPT (3.5) in answering allergological questions from clinical practice.
• To develop and apply an Allergological Error Impact Assessment to rate the severity and consequences of AI-generated errors.
• To analyze ChatGPT's accuracy, completeness, perceived humanness, and readability in allergy-related contexts.

Main Methods:

• 120 multidisciplinary allergy questions (dermatology, pediatrics, pulmonology) were posed to ChatGPT.
• Responses were assessed for accuracy, completeness, perceived humanness, and readability (Flesch Reading Ease).
• Errors were categorized by severity (minor, major, critical), with critical errors undergoing impact analysis.

Main Results:

• ChatGPT achieved good accuracy (mean 4.1/5) but exhibited critical errors in 6% of responses (1 dermatology, 2 pediatrics, 3 pulmonology).
• Completeness and perceived humanness were lower for pediatric queries.
• A critical error concerning pediatric food allergens presented a potentially life-threatening risk.

Conclusions:

• ChatGPT's current reliability in allergology is imperfect, underscoring the necessity of expert medical consultation.
• AI tools require specialized tailoring for allergy use cases to enhance their utility in clinical settings.
• Further development could enable AI like ChatGPT to safely assist in routine allergy care.