Evaluating the Quality of Artificial Intelligence Chatbot Responses in Audiology Case Interpretation and Management: A Blinded Comparative Study

This study aimed to compare the quality of audiological case interpretation and management recommendations generated by two freely-accessible large language model (LLM) chatbots (ChatGPT 3.5 and Gemini 2.0 Flash) with those of qualified audiologists, using an expert-rated quality assessment tool.

This study followed a two-phased, cross-sectional design with comparative analysis: Phase 1: 15 anonymized adult audiological cases were interpreted by 10 qualified audiologists (> 2 years' experience) and two LLM chatbots, producing 60 interpretations; each interpretation answered two questions (diagnostic interpretation and management), giving 120 responses. Phase 2: 30 qualified audiologists independently rated all 120 responses using the CLEAR (Completeness, Lack of false information, Evidence, Appropriateness, Relevance) tool to assess the quality of AI-generated health information (five raters per response; 600 total ratings), blinded to responder identity.

There were no statistically significant differences in CLEAR scores between artificial intelligence-generated and audiologist-generated responses (all p > .22 in mixed-effects modeling). Absolute differences did not exceed 1.2 CLEAR points on a 5-25 scale, with 95% confidence intervals crossing zero, indicating no statistically detectable differences in expert-rated quality. Interrater reliability was poor for single audiologist raters (ICC = .17-.18) but moderate for aggregated scores (ICC = .50-.53). Internal consistency was excellent (Cronbach's alpha = .94-.95).

Under controlled conditions, no statistically significant differences in CLEAR scores were detected between LLM-generated and audiologist-generated responses. These findings do not establish formal equivalence but represent a meaningful empirical benchmark for this emerging area of research.