Related Experiment Video
Updated: Jun 12, 2026
Digital Hybrid Model Preparation for Virtual Planning of Reconstructive Dentoalveolar Surgical Procedures
Published on: August 5, 2021
Large language models and retrieval-augmented platforms for dental implant complications: A blinded, expert-rated
Yaniv Mayer1, Luigi Canullo2, Marton Kivovics3
1The Ruth and Bruce Rappaport Faculty of Medicine, Technion - Israel Institute of Technology, Haifa, Israel; Department of Periodontology, Rambam Health Care Campus, Haifa, Israel.
Objectives:
To compare expert-rated diagnostic accuracy, patient safety, completeness of management, and freedom from detectable hallucinations across eleven conversational artificial-intelligence (AI) platforms (general-purpose large language models [LLMs], a general-purpose retrieval-augmented [RAG] platform, and a medical-domain RAG platform) using synthetic vignettes of dental implant complications.
Methods:
Thirty vignettes (10 surgical, 10 biological, 10 mechanical/prosthetic) were submitted as independent prompts to each platform. Four blinded specialists (periodontists or oral and maxillofacial surgeons, >10 years' implant experience) scored each response on a 5-point Likert scale across four parameters. Friedman tests (Kendall's W effect size) assessed ordinal scores, and Cochran's Q tested two binary outcomes: safe (Safety ≥ 4) and hallucination-free (Hallucinations ≥ 4).
Results:
Across 330 responses (1320 ratings), OpenEvidence achieved 100% safe and 100% hallucination-free responses. Among general-purpose platforms, safe rates ranged from 56.7% (Claude 4.5 Haiku) to 86.7% (Perplexity) and hallucination-free rates from 36.7% (Claude 4.5 Haiku) to 93.3% (GPT-5.4 Thinking; Qwen 3.5-Plus), both significant (Cochran's Q, p < 0.01). Friedman tests were significant for every parameter (lowest p = 1.7 × 10⁻¹⁵; Kendall's W 0.126-0.293). OpenEvidence had the highest composite score (4.90 ± 0.18), with the largest effects in the mechanical/prosthetic subcategory (W = 0.548, large).
Conclusions:
Within a vignette-based design, the medical-domain RAG platform (OpenEvidence) achieved the highest ratings across all domains, with Perplexity second; among general-purpose LLMs, GPT-5.4 Thinking, Claude Opus, and Mistral led. These findings reflect performance under tested conditions, not real-world efficacy. All platforms should serve as adjuncts to specialist judgement, and prospective validation is required before clinical deployment.
Clinical Significance:
Retrieval-augmented platforms grounded in peer-reviewed literature earned higher expert ratings than general-purpose LLMs, with the medical-domain platform OpenEvidence rated 100% safe and hallucination-free. Inter-model gaps were largest for mechanical/prosthetic cases; specialist verification remains essential before clinical use.
