From Evidence-based Endodontics to Generative AI: A Comparative Study of 11 Large Language Models.

Area of Science:

• Dental diagnostics
• Artificial intelligence in medicine
• Endodontics

Background:

• Generative large language models (LLMs) are increasingly utilized in dentistry.
• The diagnostic accuracy and reproducibility of LLMs based on clinical guidelines are not well-established.
• Authoritative position statements from the American Association of Endodontists (AAE) and European Society of Endodontology (ESE) offer evidence-based standards for evaluating LLM performance in endodontics.

Purpose of the Study:

• To assess the diagnostic accuracy and reproducibility of various generative large language models (LLMs).
• To benchmark LLM performance against established endodontic guidelines from the AAE and ESE.
• To identify variability in LLM outputs and their alignment with best practices in endodontics.

Main Methods:

• The study adhered to TRIPOD-LLM guidelines, evaluating eleven LLMs, including ChatGPT, Gemini, Claude, Perplexity, and DeepSeek models.
• Sixty multiple-choice questions derived from AAE and ESE position statements were administered to each LLM over five rounds, generating 3,300 responses.
• Primary outcome was all-correct accuracy; secondary outcome was intra-model consistency, analyzed using chi-square tests and Bonferroni adjustment.

Main Results:

• Significant variation in all-correct accuracy was observed across the eleven LLMs (χ² = 50.56, df = 10, p < 0.001).
• ChatGPT 4o and Claude Opus 4 achieved the highest accuracy at 95.0%, while DeepSeek performed lowest at 63.3%.
• Intra-model consistency generally exceeded 90% for most models, though DeepSeek showed lower consistency at 75.0%.

Conclusions:

• Most evaluated LLMs demonstrate high diagnostic accuracy and reproducibility when benchmarked against AAE and ESE endodontic guidelines.
• Despite advancements, performance variability and the potential for confidently incorrect outputs underscore the need for rigorous validation.
• Expert oversight remains crucial for the safe and effective clinical integration of LLMs in endodontics.