A comparative analysis of large language models for providing oral cavity cancer information

This study aimed to comparatively evaluate the medical information delivery capacity and content quality of current large language models (LLMs), specifically ChatGPT (GPT-5.2), Gemini (3.1), and DeepSeek (V4), regarding oral cavity cancer (OCC) based on expert opinions. 20 open-ended questions addressing the risk factors, diagnosis, and treatment of OCC were directed to the three models. The responses were evaluated using a blinded method by 31 expert physicians from Oral and Maxillofacial Surgery, Otorhinolaryngology (ENT), and Medical Oncology. The Modified Global Quality Scale (1-5 points) was utilised for evaluation. Statistical analyses were performed using Kruskal-Wallis, ANOVA, and Bonferroni post-hoc tests, while Fleiss' Kappa coefficient determined inter-expert consistency. The general performance scores of the models were high (3.57-4.15). In the overall assessment, Gemini received statistically significantly higher scores than the DeepSeek model (p = 0.036). Significant performance differences were identified across 15 of 20 questions (p < 0.05); ChatGPT excelled on clinical and treatment-oriented questions, while Gemini stood out on comprehensive informational items. While no statistically significant difference was found among the specialist groups for the overall evaluation and 19 out of 20 questions (p > 0.05), a significant difference was observed solely for Q6 (p = 0.042). Although LLMs have the potential to generate high-quality information about OCC, their performance varies by content type and model architecture. While Gemini demonstrated more consistent performance overall, expert supervision remains essential before these tools can be used as reliable sources of clinical information. Clinicians must be aware of the specific strengths and limitations of different LLMs in OCC to better guide patients who increasingly use such tools for medical information.