Performance of large language models in ambiguous treatment scenarios of early breast cancer

Consensus conferences guide treatment where evidence is ambiguous; lacking ground truth, artificial intelligence is unvalidated there. We evaluated concordance between large language models (LLMs) and expert votes at the 2025 St. Gallen International Breast Cancer Conference.

Four LLMs (Claude Opus 4.1, GPT-5, Gemini Flash 2.5, DeepSeek V3.2) each answered 81 multiple-choice questions once (324 responses). Outcomes were first-choice agreement (LLM selected the panel's modal option) and top-2 agreement (either of its two leading options; 66 questions with ≥3 options). Normalized consensus strength scored panel agreement 0-100, adjusted for option count. Mixed-effects logistic regression with random intercepts for question modelled agreement on consensus strength, complexity and LLM (Model 1: 324 observations/81 questions; Model 2: 264/66). Agreement was also compared with an option-count-weighted null (permutation, question-level cluster bootstrap). Questions with ≤2 LLMs matching the panel underwent qualitative review.

First-choice agreement was 54-64% and top-2 agreement 77-89%. Agreement exceeded chance overall (58.6% vs 29.8%; +28.9 points, 95% CI 21.4-36.4) and for questions with 3, 4 and ≥ 5 options (42.6-66.1% vs 18.8-33.3% expected), but not binary questions (65.0% vs 50.0%; +15.0, 95% CI -5.0 to 33.3). Consensus strength was the predominant factor associated with concordance (Model 1: odds ratio 1.04/point, 95% CI 1.02-1.06, P < 0.001; Model 2: 1.06, 1.03-1.09, P < 0.001). Qualitative review identified knowledge gaps in 18 questions (22%), defensible alternatives in 14 (17%) and evidence-consistent insights in 8 (10%).

LLMs amplify prevailing evidence but show significant limitations in ambiguous scenarios. Prospective evaluation within consensus processes is needed.