Implementing a context-augmented large language model to guide precision cancer medicine.

Area of Science:

• Oncology
• Medical Informatics
• Artificial Intelligence

Background:

• Precision cancer medicine relies on molecularly informed therapies and FDA approvals, posing integration challenges for oncologists.
• Large Language Models (LLMs) show clinical potential but lack specialized knowledge for up-to-date oncology treatment recommendations.

Purpose of the Study:

• To develop and evaluate a Retrieval-Augmented Generation-LLM (RAG-LLM) workflow for accurate biomarker-driven cancer therapy recommendations.
• To compare the RAG-LLM approach against an LLM-only framework using both structured and unstructured data.

Main Methods:

• Developed a RAG-LLM workflow incorporating the Molecular Oncology Almanac (MOAlmanac) knowledge resource.
• Evaluated performance on 234 therapy-biomarker relationships and real-world oncologist queries.
• Compared RAG-LLM with structured and unstructured data augmentation against an LLM-only approach.

Main Results:

• RAG-LLM achieved 79-95% accuracy, significantly outperforming LLM-only (62-75%).
• Structured data augmentation boosted precision (49% to 80%) and F1-score (57% to 84%) compared to unstructured data.
• RAG-LLM demonstrated 81-90% accuracy on practicing oncologists' queries.

Conclusions:

• The RAG-LLM framework effectively provides precise, reliable FDA-approved precision oncology therapy recommendations.
• Integrating a curated, structured knowledge base like MOAlmanac is crucial for enhancing LLM performance in oncology.
• Further development is needed to address ambiguous clinical scenarios.