RadOncRAG: A Novel Retrieval-Augmented Generation Framework Improves Large Language Model Benchmark Performance in...

Area of Science:

• Artificial Intelligence in Medicine
• Oncology
• Medical Education

Background:

• Large language models (LLMs) show potential in oncology but face challenges like hallucinations and outdated data.
• Retrieval-augmented generation (RAG) addresses these limitations by integrating external, domain-specific knowledge.

Purpose of the Study:

• To evaluate the impact of a RAG pipeline on the performance of various LLMs in radiation oncology.
• To compare zero-shot LLM performance against RAG-augmented performance using a radiation oncology examination dataset.

Main Methods:

• Fifteen LLMs were tested on 298 questions from the 2021 American College of Radiology in-training examination.
• A RAG pipeline (Iridium Model) was implemented, querying a specialized radiation oncology database to augment prompts.
• Performance was compared between zero-shot and RAG-augmented workflows.

Main Results:

• Larger LLMs demonstrated higher zero-shot accuracy, with some outperforming graduating residents.
• Reasoning models like GPT-4o achieved high accuracy without RAG; RAG did not improve their performance.
• RAG enhanced performance for non-reasoning models, with domain-specific gains in clinical, biology, and physics knowledge areas.
• Majority voting improved aggregate accuracy, while RAG and reasoning models increased computational costs.

Conclusions:

• A radiation oncology-specific RAG pipeline boosts non-reasoning LLM performance by incorporating domain-specific evidence.
• RAG does not enhance the performance of advanced reasoning models in this context.
• RAG provides an efficient, cost-effective alternative to extensive model training for clinical decision support, offering citable explanations.