Performance of large language models on the Turkish Pharmacy Specialty Examination: a comparative analysis of...

Large language models (LLMs) have demonstrated strong performance in answering knowledge-based questions in healthcare education. Specialty examinations offer a standardized and objective framework to assess these capabilities. However, to date, no study has evaluated LLM performance on the Turkish Pharmacy Specialty Examination (EUS), a nationally standardized exam applied for the purpose of admitting candidates to pharmacy specialization programs. Therefore, this study aimed to comparatively evaluate LLM performance on EUS questions in terms of accuracy, self-reported confidence, and readability. This study conducted a comparative evaluation of three LLMs-ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash-using publicly available 84 multiple-choice questions from the EUS between 2017 and 2025. Each question was submitted to each model in a separate, newly initiated session using a standardized prompt. Model performance was assessed based on answer accuracy, self-reported confidence (1-5 scale), and readability of generated responses, using the Flesch reading ease (FRE), gunning fog index (GFI), and simple measure of Gobbledygook (SMOG) indices. All statistical analyses were performed using non-parametric repeated-measures methods, including Cochran's Q test for paired categorical comparisons and the Friedman test with Durbin-Conover post-hoc analyses for readability scores, with two-tailed significance set at p < 0.05. Overall, the evaluated LLMs exhibited high performance. Gemini 2.5 Flash achieved the highest overall accuracy rate (92.9%), followed by ChatGPT-5.1 (90.5%) and DeepSeek-R1 (89.3%), with no statistically significant difference among the models (p = 0.584). Self-reported confidence was predominantly maximal (5/5), with ChatGPT-5.1, DeepSeek-R1, and Gemini 2.5 Flash assigning maximum confidence to 87.5, 55.6, and 66.7% of incorrect responses, respectively. Significant differences in readability were observed among the evaluated LLMs. ChatGPT-5.1 generated texts with lower GFI and SMOG scores compared with DeepSeek-R1 and Gemini 2.5 Flash (p < 0.05), indicating lower linguistic complexity. No statistically significant differences were identified among models for FRE. LLMs demonstrated high and comparable accuracy when answering domain-specific pharmacy examination questions; however, occasional overconfidence in incorrect responses highlights the need for careful oversight. Differences in linguistic complexity underscore the importance of selecting models optimized for readability in educational settings. Overall, these findings suggest that LLMs may have potential as supplementary tools in pharmacy education within examination-based contexts, provided that expert guidance and critical appraisal are maintained to ensure reliability and clarity.