Related Experiment Video
Updated: Oct 20, 2026
A Postoperative Evaluation Guideline for Computer-Assisted Reconstruction of the Mandible
Published on: January 28, 2020
Evaluating different large language model performances in evidence-based dental implantology: a blinded controlled
Vasilios Alevizakos1,2, Robert Gaudin3, Iman Ghanad3
1Research Center for Digital Technologies in Dentistry and CAD/CAM, Department of Dentistry, Faculty of Medicine and Dentistry, Danube Private University, Steiner Landstrasse 124, Krems an der Donau, 3500, Austria. vasileios.alevizakos@googlemail.com.
Background:
Artificial intelligence (AI) and large language models (LLMs) are increasingly being explored for clinical decision-making and evidence-based information retrieval in dental implantology. However, concerns remain regarding the accuracy, reliability, and clinical applicability of different AI approaches, including general-purpose, fine-tuned, and retrieval-augmented generation (RAG) models. This study aimed to compare their ability to provide accurate, evidence-based, and clinically relevant responses to implantology-related questions.
Methods:
Sixteen master's-level clinical questions were presented to all four LLMs under controlled conditions. Their anonymized responses were independently evaluated by three dental implantology specialists based on three core criteria: specificity, accuracy, and clinical utility, using a 0-10 scale. Descriptive statistics were reported as medians and interquartile ranges. Differences among models were assessed primarily using Kruskal-Wallis tests, and inter-rater reliability was evaluated using Fleiss' kappa.
Results:
LLM4 consistently achieved the highest median scores across all criteria (Specificity: median 9.0, IQR 9.0-10.0; Accuracy: median 9.5, IQR 9.0-10.0; Clinical Utility: median 10.0, IQR 9.0-10.0), significantly outperforming the other models (all p < 0.001). Cohen's d revealed large to very large effect sizes in comparisons involving LLM4, particularly against LLM2 and LLM3. LLM2 scored lowest and showed the highest variability. Inter-rater reliability was slight across all metrics (κ = 0.078-0.180), indicating moderate evaluator disagreement.
Conclusions:
Among the models tested, LLaMA 3.2 + RAG demonstrated superior and more consistent clinical performance in dental implantology. While fine-tuning provided moderate benefits, real-time access to structured guidelines significantly enhanced response quality. Despite low inter-rater agreement, these findings highlight the promise of RAG models in supporting high-complexity, evidence-based clinical domains, while emphasizing the need for standardization in AI evaluation protocols.