Can artificial intelligence chatbots think like dentists? A comparative analysis based on dental specialty...

The integration of artificial intelligence (AI) in healthcare and medical education has advanced rapidly, with conversational AI systems gaining attention for their potential in academic assessment and clinical reasoning. This study aimed to evaluate AI chatbots' performance on restorative dentistry questions from the Turkish Dental Specialty Examination (DUS), a high-stakes national exam assessing theoretical and clinical knowledge.

An in silico, cross-sectional, comparative design was employed. A total of 190 multiple-choice questions (MCQs) from 19 DUS sessions between 2012 and 2025 were obtained from the Assessment, Selection, and Placement Center (ÖSYM) website. After excluding annulled items, 188 questions were analyzed. Eight AI chatbots (ChatGPT-3.5, ChatGPT-4o Free, ChatGPT-4o Plus, Claude Sonnet 4, Microsoft Copilot, DeepSeek, Gemini 1.5, and Gemini Advanced) were tested using a standardized single-attempt protocol in Turkish. Performance measures included accuracy, response length, and response time. Questions were categorized by year, content domain, and length for subgroup analyses. Statistical analyses were conducted in Python using standard libraries. Descriptive statistics and Pearson's correlation were calculated, while comparisons involved the Shapiro-Wilk test, Levene's test, Kruskal-Wallis test, and Dunn's post hoc test, with significance set at p < 0.05.

No significant difference was found in overall accuracy (p = 0.18). However, response time and word count differed significantly (p < 0.001). Gemini Advanced showed the highest accuracy (96.28%), followed by ChatGPT-4o Plus (93.62%). Gemini 1.5 produced the longest yet fastest responses, while DeepSeek had the lowest accuracy and slowest responses. Accuracy remained stable across years but varied by topic, with lower performance in complex areas such as cavity preparation. In case-based questions, Gemini Advanced, Gemini 1.5, and ChatGPT-4o Plus achieved 100% accuracy. Performance in image-based questions was inconsistent, underscoring limitations in visual reasoning.

AI chatbots demonstrated high accuracy in answering restorative dentistry exam questions, with Gemini Advanced, ChatGPT-4o Plus, and Gemini 1.5 showing superior performance. Despite differences in response time and content length, their potential as supplementary tools in dental education is evident, warranting further validation across specialties and contexts.

Not applicable.