Automated Prediction of Radiological Protocols Using Retrieval Augmented Generation.

Area of Science:

• Artificial Intelligence in Radiology
• Clinical Decision Support Systems
• Medical Informatics

Background:

• Radiological protocol selection is complex, time-consuming, and prone to errors.
• Automated methods face challenges like class imbalance and site-specific variations.
• Large language models (LLMs) offer potential for improving protocol selection efficiency.

Purpose of the Study:

• To evaluate the efficacy of LLMs for large-scale radiological protocol selection.
• To compare retrieval-augmented generation (RAG) against direct fine-tuning for protocol selection.
• To assess the impact of RAG on accuracy, abstention rates, and adaptability across different clinical sites.

Main Methods:

• Trained site-specific Llama 3.2 3B LLMs using patient reports from three Mayo Clinic sites.
• Implemented RAG by integrating division-scoped FAISS indexes for contextual evidence.
• Compared performance of fine-tuned non-RAG and RAG-augmented models using macro and weighted F1 scores.

Main Results:

• Both RAG and non-RAG models demonstrated strong baseline performance across sites.
• RAG significantly improved macro F1 at Arizona and Florida sites but not Rochester.
• RAG introduced an interpretable abstention mechanism with low baseline rates (1-2.5%) at most sites.

Conclusions:

• LLMs, particularly with RAG, can support reliable radiological protocol selection at scale.
• RAG effectiveness is heterogeneous across sites, necessitating site-specific tuning.
• RAG's adaptable retrieval indexes and abstention mechanism offer operational advantages for clinical workflow integration.