Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study

Aim: Large language models (LLMs) may facilitate the interpretation of clinical guideline information, but their performance in renal cell carcinoma (RCC) remains insufficiently characterized. We evaluated agreement between a multidisciplinary expert panel and the responses generated by Claude, Gemini and ChatGPT to RCC guideline-based questions. Materials & methods: A multidisciplinary tumor board developed 25 clinical questions based on the 2025 European Association of Urology RCC guidelines. Guideline figures and tables were converted into structured, plain-language textual summaries and supplied to the models. Each LLM generated one response per question using direct prompting. An 11-member multidisciplinary expert panel evaluated the responses in a two-round modified Delphi process using a 9-point scale and predefined consensus criteria. Results: Consensus in agreement was reached for 24/25 Gemini responses (96%) and 23/25 responses (92%) for both Claude and ChatGPT. No response reached consensus in disagreement. The proportion of responses with a median score ≥7 was 25/25 (100%) for Gemini, 24/25 (96%) for Claude and 23/25 (92%) for ChatGPT. No statistically significant differences were detected among the models overall or in pairwise comparisons. Responses requiring further discussion or failing to reach consensus mainly involved clinically uncertain or individualized scenarios. Panelists identified occasional factual inaccuracies, omitted options and limited clinical nuance. Conclusion: When provided with structured, guideline-derived material, selected LLMs generated responses that frequently aligned with multidisciplinary expert consensus. These findings support their potential for guideline interpretation but do not demonstrate independent clinical reasoning or real-world decision-making performance. Expert oversight remains essential, particularly in complex scenarios.