Are AI chatbots ready for endodontics? Evaluating their validity, consistency, and readability in patient-oriented...

This study aimed to evaluate and compare the responses provided by ChatGPT-4o, Google Gemini (2.0 Flash) and Microsoft Copilot to frequently asked questions (FAQs) in endodontics. Fifty patient-oriented, open-ended endodontic FAQs were formulated by two experienced endodontists. Each question was posed to each chatbot three times, yielding a total of 450 responses. Two endodontists independently evaluated all responses using a modified Global Quality Score (GQS) on a five-point Likert scale. Validity was evaluated at two thresholds: low (all three responses scored ≥ 4) and high (all three responses scored 5). Fisher's exact test was used to compare validity among models. Cronbach's alpha was calculated to assess consistency. Readability was analyzed using the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). All chatbots performed well under the low-validity threshold, but their performance declined under the stricter high-validity criteria. High-threshold validity of Google Gemini was significantly greater than that of ChatGPT-4o (p = 0.022). Unlike the other chatbots, ChatGPT-4o did not receive any scores of 2, yet its mean overall score was significantly lower than those of the other two models (p < 0.05). This suggests that models generating more detailed responses may carry a higher risk of misleading information, even when achieving higher scores. By contrast, ChatGPT-4o produced more readable outputs, which may lack sufficient depth. Overall, these findings indicate that no single chatbot can be considered optimal across all dimensions, as readability, accuracy, and completeness cannot be fully achieved by one model alone.