Clinical evaluation of large language model recommendations in melanoma: comparison with multidisciplinary tumor board decisions in a real-world cohort

Large language models (LLMs) are increasingly being studied as potentially valuable support tools in oncology practice including clinical decision support. Yet, their real-world utility in melanoma treatment decision-making is still not sufficiently considered, especially in resource-limited settings. Accordingly, this study evaluated the performance of four LLMs against real-world treatment decisions of a melanoma multidisciplinary tumor board (MDT).

This retrospective single-center study included 151 consecutive patients with newly diagnosed cutaneous melanoma discussed at the MDT at the University Clinical Center Tuzla, Bosnia and Herzegovina, between 2020 and 2024. Melanoma treatment recommendations generated by four LLMs, ChatGPT-4o, ChatGPT-5 Thinking, Gemini 2.5 Pro and DeepSeek-V3.2, were evaluated by four board-certified oncologists against the actual MDT treatment decisions. Additionally, the LLM-generated recommendations were also rated across five pre-specified domains: clarity, clinical applicability, coverage, explanation and support with evidence, and guideline concordance.

In this study, inter-rater reliability was acceptable to good, supporting the consistency of expert evaluation. ChatGPT-5 Thinking showed the strongest and most consistent overall performance, followed by ChatGPT-4o, while Gemini 2.5 Pro and DeepSeek-V3.2 were rated less favorably. Differences between LLMs were statistically significant across all evaluated domains. Performance differences appeared most clinically relevant in more complex scenarios, particularly when consideration of adjuvant or systemic treatment strategies was required.

The findings of this study suggest that selected LLMs may have a supportive role in everyday melanoma MDT practice particularly in oncology centers with limited resources. However, the current results do not support the use of LLM-generated recommendations as independent treatment decisions, and further prospective studies are required before LLM-assisted treatment recommendations can be safely integrated into the MDT workflow.