Performance of Retrieval-Augmented Generation Large Language Models in Guideline-Concordant Prostate-Specific Antigen...

Area of Science:

• Urology
• Medical Informatics
• Artificial Intelligence in Medicine

Background:

• Prostate-specific antigen (PSA) testing is crucial for early prostate cancer detection.
• Adherence to clinical guidelines for PSA screening is challenging, leading to high rates of inappropriate testing.
• Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding them in trusted external data sources.

Purpose of the Study:

• To evaluate a RAG-enhanced LLM system for its effectiveness in providing guideline-concordant PSA screening recommendations.
• To compare the performance of the RAG-LLM system against junior clinicians in a simulated clinical setting.

Main Methods:

• Developed a RAG pipeline integrating European and American urology guidelines for PSA testing.
• Created 44 fictional outpatient case scenarios for testing.
• Compared RAG-LLM recommendations against those from five junior clinicians (closed-book and open-book formats).

Main Results:

• The RAG-LLM tool achieved 95.5% guideline-concordant recommendations.
• Junior clinicians achieved 62.3% (closed-book) and 74.1% (open-book) accuracy.
• The RAG-LLM system demonstrated statistically significant improvement over clinicians (P<.001).

Conclusions:

• RAG techniques enable LLMs to effectively integrate complex medical guidelines.
• RAG-LLM tools can enhance clinical decision-making in urology for PSA testing.
• This technology has the potential to improve healthcare consistency and reduce clinician cognitive load.