Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Oncology Informatics

Background:

• The dynamic nature of breast cancer treatment presents challenges for clinicians in synthesizing up-to-date information.
• Proprietary large language models (LLMs) offer potential but face limitations in cost, privacy, and accessibility.
• Open-source LLMs present an alternative for developing specialized clinical support tools.

Purpose of the Study:

• To evaluate the performance of small, open-source LLMs augmented with retrieval-augmented generation (RAG) for breast cancer clinical guideline queries.
• To compare the performance of RAG-enhanced open-source LLMs against state-of-the-art proprietary models.
• To assess the feasibility of using these models for clinical decision support.

Main Methods:

• A domain-specific RAG pipeline was developed using 1,356 ASCO breast cancer guideline documents.
• Five LLMs (GPT-4-turbo, GPT-3.5-turbo, Qwen2.5-14B, LLaMA3-8B, OpenBioLLM-8B) were tested with and without RAG.
• Performance was evaluated using expert-curated question-answer triplets and rubric-based scoring, with GPT-4-turbo as judge and human oncologist validation.

Main Results:

• RAG-enhanced Qwen2.5-14B demonstrated performance comparable to GPT-4-turbo, with relative improvements in win rates of 16% to 46%.
• Absolute gains in rubric scores were modest, but RAG consistently improved LLM performance.
• Human expert validation confirmed the superiority of RAG but yielded more conservative scores than LLM judges.

Conclusions:

• Optimized RAG with small open-source LLMs can achieve performance close to proprietary models for clinical decision support.
• This approach offers a scalable, cost-effective, and privacy-preserving solution for clinical implementation.
• Potential for deployment on single-GPU infrastructure under expert supervision exists.