Retrieval Augmented Therapy Suggestion for Molecular Tumor Boards: Algorithmic Development and Validation Study.

Area of Science:

• Oncology
• Medical Informatics
• Artificial Intelligence

Background:

• Molecular tumor boards (MTBs) involve extensive manual review for patient treatment recommendations.
• Large language models (LLMs) offer potential to enhance MTB efficiency, reduce errors, and improve care accessibility in precision oncology.

Purpose of the Study:

• To evaluate the effectiveness of LLM-generated treatment suggestions for MTB patients.
• To assess LLMs' capability in providing evidence-based recommendations with PubMed citations.

Main Methods:

• A retrieval augmented generation pipeline was developed using PubMed data.
• LLMs were prompted to generate treatment recommendations with PubMed references for a test patient cohort.
• MTB members manually reviewed the LLM-generated recommendations for accuracy and relevance.

Main Results:

• 75% of cited articles were correctly sourced from PubMed; 17% were identified as hallucinations.
• Clinician-evaluated LLM queries showed higher accuracy than automated ones.
• Clinicians found 25% of LLM responses equivalent to their own and 37.5% as plausible alternatives.

Conclusions:

• LLM-enhanced retrieval augmented generation can accelerate MTB processes, sometimes yielding clinician-equivalent recommendations.
• Further research is needed to eliminate hallucinations and ensure consistent LLM performance.
• LLMs offer a scalable solution for MTB investigations but require significant clinician oversight and cannot fully automate the process.