Performance of large language models in a high-stakes dental assessment: evidence from the Turkish dentistry...

Large language models (LLMs) are increasingly used in dental education; however, their performance on high-stakes national examinations remains insufficiently explored. The Dentistry Specialization Examination (DUS) in Türkiye is a standardized gateway to postgraduate dental training and includes a dedicated periodontology component. No previous study has evaluated contemporary LLM accuracy on DUS periodontology questions or examined whether Bloom's Revised Taxonomy predicts model performance. A total of 128 validated periodontology single-best-answer multiple-choice questions from publicly released DUS booklets (2012-2021) were analyzed. Each question was presented in Turkish to ChatGPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, and DeepSeek v3.2 using a standardized single-answer prompt. Responses were scored against the official key. Two periodontology specialists independently classified items according to Bloom's Revised Taxonomy and eight periodontal content domains. Accuracy differences were assessed using Cochran's Q test and Bonferroni-adjusted McNemar tests. The association between Bloom level and accuracy was examined using binary logistic regression. Accuracy rates ranged from 80.47% to 91.41% across the four large language models. Incorrect responses were distributed across content domains, with a greater number observed in the non-surgical and surgical periodontal therapy domains. Regarding cognitive demand, inter-rater reliability for Bloom classification was high (κ = 0.948); however, Bloom level was not a statistically significant predictor of model accuracy (p > 0.05). Differences in performance across models were observed; a statistically significant difference was identified only between Gemini 2.5 Pro and DeepSeek v3.2 (p = 0.0081). Contemporary LLMs showed high accuracy on released DUS periodontology single-best-answer multiple-choice items. While performance was strong in this structured, text-based examination context, the ability of these models to replicate higher-order clinical reasoning remains uncertain. Within the context of the analyzed DUS periodontology dataset, these findings contribute to ongoing discussions regarding the design and potential vulnerabilities of high-stakes single-best-answer MCQ-based assessments in the era of artificial intelligence. Consequently, it may be important for educators to understand the potential role of LLMs in students' preparation for assessments, suggesting that future research may explore modifications in test development to better capture student understanding.