Related Experiment Video
Updated: Jun 26, 2026
Comparison of Predictive Performance of Three Lymph Node Staging Systems in Colorectal Signet Ring Cell Carcinoma Based on Machine Learning Model
Published on: April 18, 2025
Assessment of Large Language Models in Colorectal Cancer Multidisciplinary Tumor Board Decision-Making: A
Aydan Farzaliyeva1, Mehmet Nezir Ramazanoglu1, Arzu Oguz1
1Division of Medical Oncology, Department of Internal Medicine, Faculty of Medicine, Baskent University, 06490 Ankara, Türkiye.
Background:
Large language models (LLMs) are emerging as clinical decision-support tools in oncology, yet their ability to generate reliable treatment recommendations in real-world multidisciplinary tumor board (MTB) settings remains uncertain, particularly for complex colorectal cancer (CRC).
Methods:
In this retrospective study, 300 consecutive adult CRC cases discussed at a tertiary MTB were evaluated. Standardized de-identified case summaries were independently submitted to Gemini 2.5 (general-purpose, guideline-integrated) and MedGemma 27B (domain-specialized; T = 0.0 and T = 1.0). Concordance with MTB decisions was assessed using weighted Cohen's kappa (κ), accuracy, F1 score, and recall. Safety was adjudicated by blinded senior MTB members using a three-tier risk framework. Importantly, Gemini 2.5 was evaluated in a guideline-integrated setting, whereas MedGemma operated without external guideline retrieval, introducing a predefined asymmetry in knowledge augmentation.
Results:
Gemini 2.5 demonstrated substantial agreement (κ = 0.792, p < 0.001), the highest exact concordance (85.0%), and a clinical safety rate of 83.7%, significantly outperforming MedGemma (p < 0.001). MedGemma showed moderate agreement at T = 0.0 (κ = 0.566; accuracy = 70.3%) and modest improvement at T = 1.0 (κ = 0.610; accuracy = 73.5%; p = 0.038), with lower safety rates (62.3% and 62.7%; p < 0.001). Discordance predominantly occurred in clinically complex scenarios, including surgical/interventional planning, active surveillance, recurrent disease management, and patients with compromised performance status.
Conclusions:
A guideline-integrated general-purpose LLM demonstrated superior concordance and safety compared with a domain-specialized model operating without external retrieval, supporting its adjunctive use within MTBs while preserving expert clinical judgment.
