Expert Evaluation of Artificial Intelligence Chatbots for Central Auditory Processing Disorder Information

Artificial intelligence (AI) chatbots based on large language models (LLMs) can deliver medical information, but their performance on specialized topics such as central auditory processing disorder (CAPD) remains unexplored. This study evaluated the accuracy and completeness of three AI chatbots (ChatGPT, Gemini, and Claude) in providing CAPD-related information across varying levels of question complexity.

Forty-four questions, categorized into four difficulty levels (patient level, easy, intermediate, and specialized; n = 11 each), were submitted to each chatbot, generating 132 responses. Seven clinical experts, blinded to chatbot identity, independently rated accuracy and completeness on a 1-5 Likert scale. Data were analyzed with analyses of variance, correlations, and interrater comparisons.

Chatbot performance was similar, with mean accuracy below 4.0 and completeness about 3.5. Complex questions often scored below 3.0 across experts. Only three of the 44 questions, primarily patient level or relatively simple, received consistently high expert ratings (≥ 4 for both accuracy and completeness) across all three chatbots. Performance declined with question difficulty, although differences were not statistically significant. Accuracy and completeness were correlated across chatbots.

Current AI chatbots provided generally accurate CAPD information but fell short of clinical standards, particularly on specialized questions. Their limited performance underscores the need for clinician oversight in CAPD assessment and management. Chatbots may serve as helpful adjuncts but should not replace expert evaluation and guidance in clinical settings.

https://doi.org/10.23641/asha.31975101.