Prediction of tumor board procedural recommendations using large language models

Area of Science:

• Oncology
• Artificial Intelligence in Medicine
• Medical Decision Support

Background:

• Multidisciplinary tumor boards enhance cancer patient survival but are resource-intensive.
• Evaluating AI for procedural recommendations in oncology is crucial for improving efficiency.

Purpose of the Study:

• To assess the accuracy and quality of large language models (LLMs) in recommending procedures for Head and Neck Oncology tumor boards.
• To compare parameter-efficient fine-tuning (PEFT) and in-context learning (ICL) for adapting LLMs to this task.

Main Methods:

• LLMs were adapted using PEFT and ICL on a dataset of 229 training and 100 validation records.
• Human experts conducted randomized, blinded classifications to evaluate model predictions.
• Model performance was measured by treatment line congruence and the medical justifiability of recommendations.

Main Results:

• LLM recommendations achieved up to 86% treatment line congruence and 98% medical justifiability.
• PEFT outperformed ICL, and larger commercial models generally showed better results.
• Model performance varied, highlighting the need for further optimization.

Conclusions:

• LLMs can feasibly provide precise, medically justifiable procedural recommendations for oncology patients.
• Expanding datasets and incorporating updated guidelines can improve LLM factuality and guideline alignment.
• Further research is encouraged to enhance LLM reliability in medical decision-making.