Related Experiment Video
Updated: May 20, 2026
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
Evaluating reasoning models for therapy recommendations in gastrointestinal stromal tumors: expert and LLM-based
Melissa Harbrücker1,2, Franka Menge1, Arshia Taebi1
1Department of Surgery, Medical Faculty Mannheim, University Medical Center Mannheim, University of Heidelberg, Mannheim, Germany.
Purpose:
This study aims to evaluate two advanced reasoning LLMs in generating treatment recommendations for real-world gastrointestinal stromal tumor (GIST) cases and assess their concordance with multidisciplinary team (MDT) decisions at a certified tertiary sarcoma center.
Methods:
Sixty-five real-world GIST cases from a tertiary sarcoma center were used to compare two advanced reasoning models-OpenAI o1 and DeepSeek-R1. Recommendations were generated using a multi-expert prompting strategy with current clinical guidelines as context. Five sarcoma specialists and an independent LLM (Mistral AI) evaluated alignment with MDT decisions and guideline concordance.
Results:
OpenAI o1 achieved higher concordance with MDT decisions than DeepSeek-R1 (76.9% vs. 53.8%, p < 0.001) and more recommendations aligned with either MDT or guidelines (80.0% vs. 63.1%, p = 0.005). Inter-rater reliability among human evaluators was excellent (ICC = 0.929). The LLM judge's evaluations showed moderate agreement with human assessments (κ = 0.647). OpenAI o1 responses were significantly longer than those of DeepSeek-R1 and MDT records.
Conclusions:
OpenAI o1 outperformed DeepSeek-R1 in generating clinically relevant GIST therapy recommendations. The study highlights the feasibility of using LLMs both as decision support tools and as evaluators ("LLM-as-a-judge") in oncology, while emphasizing the need for expert oversight in clinical deployment.
