Consistency and accuracy of different artificial intelligence models in evaluating longitudinal cracks and fractures...

Large language models such as ChatGPT and Google Gemini have attracted increasing attention in dentistry due to their potential to support clinical decision-making; however, their reliability in responding to endodontic questions remains unclear. This study compared the accuracy, consistency, and temporal stability of three artificial intelligence models (ChatGPT-3.5, ChatGPT-4o, and Google Gemini) when answering dichotomous clinical questions on vertical root fractures and tooth cracks using expert responses as the reference standard. Seventy-two true/false questions were developed based on the European Society of Endodontology position statement on longitudinal tooth fractures, and 60 validated items with an item-level content validity index of 1.00 were included and categorized into easy, moderate, and difficult levels. Each model was queried over 10 consecutive days at three daily time points, generating 5,400 responses in total. Overall accuracy differed significantly among models (p < 0.001), with ChatGPT-4o achieving the highest accuracy (86.1%), followed by ChatGPT-3.5 (85.6%) and Gemini (81.8%). No differences were observed between morning and afternoon sessions, whereas evening performance was significantly lower for Gemini (p = 0.011). Differences between models were evident only for moderate-difficulty questions (p < 0.001). Inter-model agreement was very low (Cohen's kappa (κ) = 0.019), and temporal stability differed significantly among models (p < 0.001), with ChatGPT-4o showing the highest stability. Although acceptable accuracy levels were observed, limited agreement and temporal variability indicate that these systems cannot replace clinician judgment and should currently be used only as supportive tools in endodontic decision-making.