Related Experiment Video
Updated: Apr 18, 2026
07:14
Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
Published on: December 23, 2025
899
Expert Evaluation of Artificial Intelligence Chatbots for Central Auditory Processing Disorder Information
Alyssa J Davidson1, W Wiktor Jedrzejczak2,3, Jennifer McCullagh4
1Department of Audiology and Speech Language Pathology, Walter Reed National Military Medical Center, Bethesda, MD.
American Journal of Audiology
|April 16, 2026
Summary
Artificial intelligence (AI) chatbots provide generally accurate information on central auditory processing disorder (CAPD) but do not meet clinical standards for complex queries. Clinician oversight remains essential for CAPD assessment and management.
Area of Science:
- Medical Informatics
- Artificial Intelligence in Healthcare
- Audiology
Background:
- Large language models (LLMs) power AI chatbots capable of disseminating medical information.
- The performance of these AI chatbots on specialized medical topics, such as central auditory processing disorder (CAPD), is largely unexplored.
- Evaluating AI chatbot accuracy and completeness is crucial for understanding their potential role in healthcare.
Purpose of the Study:
- To assess the accuracy and completeness of three leading AI chatbots (ChatGPT, Gemini, and Claude) in providing information on central auditory processing disorder (CAPD).
- To analyze chatbot performance across varying question complexities, from patient-level to specialized inquiries.
- To establish a benchmark for AI chatbot performance in a specialized audiology context.
Main Methods:
- Forty-four CAPD-related questions were developed, categorized into four difficulty levels (patient, easy, intermediate, specialized).
- Each question was submitted to ChatGPT, Gemini, and Claude, yielding 132 responses.
- Seven independent clinical experts evaluated response accuracy and completeness using a 5-point Likert scale, with data analyzed via ANOVA and correlation.
Main Results:
- All three AI chatbots demonstrated similar performance, with average accuracy below 4.0 and completeness around 3.5.
- Responses to complex and specialized questions frequently scored below 3.0.
- Only three questions, mostly patient-level or simple, received consistently high expert ratings (≥ 4) across all chatbots.
- Performance generally declined with increased question difficulty, though not always statistically significant.
Conclusions:
- Current AI chatbots offer generally accurate CAPD information but fall short of clinical standards, especially for complex topics.
- The findings highlight the necessity of clinician supervision in CAPD assessment and management.
- AI chatbots can serve as supplementary tools but should not substitute expert clinical evaluation and guidance.
