Challenges in AI Based Tumor Board Case Summarization and Recommendations

Tumor boards, recurring meetings at hospital institutions, assemble multiple cancer specialties (e.g. medical oncology, pathology) to discuss oncological care for patients. In this work, we formally describe the tasks of tumor board case summarization, options generation, and meeting outcomes prediction as AI problems. We study datasets from 4+ medical institutions, providing the performance of available state-of-the-art large language models (LLMs) across DeepSeek, GPT, Gemini, Qwen families, with human-expert evaluations. In total, medical oncologists created reference texts and human judgments across tasks totaling ~10k and ~13k respectively. Results revealed clinicians rated LLM-generated case summaries 3.57-4.59 out of 5.0 across institutions, but struggled with the recommendations generation task, averaging 2.0-3.6. Clinical alignment studies revealed modest correlations of 0.2 for case summarization subtasks, but higher for recommendations at 0.6 for LLM-as-Judge metrics. Our proposed TBFact metrics were shown to be competitive with LLM-as-Judge metrics in longer free text settings, suggesting a promising direction towards explainable metrics. This work, the largest study with expert grading with both granular and direct assessments on multiple axes (e.g. completeness, factual accuracy), revealed surprising nonlinear relationships between granular criterion-level scores and overall direct assessments. Finally, our study revealed experts frequently assigned high ratings to responses that differed substantially from their own reference texts, challenging the core assumptions underlying reference-based automatic evaluation metrics.