Evaluating Adherence to Canadian Radiology Guidelines for Incidental Hepatobiliary Findings Using RAG-Enabled LLMs.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Radiology Decision Support

Background:

• Large language models (LLMs) show potential for clinical decision support.
• Current LLMs often lack up-to-date clinical guideline integration.
• Retrieval-augmented generation (RAG) offers a method to dynamically incorporate external information.

Purpose of the Study:

• To evaluate the performance of GPT-4o and o1-mini LLMs in adhering to Canadian radiology guidelines.
• To assess the impact of RAG on guideline adherence for incidental hepatobiliary findings.
• To compare RAG-enabled LLMs against their non-RAG counterparts.

Main Methods:

• A custom RAG architecture was developed to integrate guideline recommendations.
• Clinical cases (319) were used to prompt LLMs with and without RAG.
• Guideline adherence rates, reading ease, grade level, and response times were analyzed.

Main Results:

• RAG significantly improved adherence rates for both GPT-4o (81.7% to 97.2%) and o1-mini (79.3% to 95.1%).
• RAG-enabled models showed better reading ease and lower grade level scores.
• Response times increased slightly with RAG but remained clinically acceptable.

Conclusions:

• RAG-enabled LLMs substantially enhance adherence to radiology guidelines for hepatobiliary findings.
• This approach shows promise for improving evidence-based care in clinical settings.
• Further validation of RAG in broader clinical applications is warranted.