Locally deployed context-aware chatbot outperforms generic large language models for guideline-concordant pediatric...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Clinical Decision Support Systems
• Pediatric Radiology

Background:

• Accurate modality selection in pediatric imaging is crucial but challenging.
• Adherence to American College of Radiology (ACR) Appropriateness Criteria is often limited.
• Large language models (LLMs) show promise for decision support but require domain-specific accuracy.

Purpose of the Study:

• To assess the performance of a locally run, context-aware chatbot (ped-Llama) for pediatric imaging recommendations.
• To evaluate its accuracy against ACR guidelines and compare it with other LLMs and human experts.
• To determine the utility of retrieval-augmented generation (RAG) in enhancing LLM performance for medical applications.

Main Methods:

• A simulation study used 50 pediatric clinical scenarios based on ACR guideline variants.
• The ped-Llama chatbot, utilizing a RAG approach with Llama-3.1-8B, was compared to generic LLMs (Llama-3.1-8B, GPT-4o, Claude Opus) and radiologists.
• Recommendations were classified using ACR criteria, and LLM output consistency was analyzed.

Main Results:

• ped-Llama achieved 80% "Usually appropriate" recommendations, surpassing generic LLMs and matching specialist radiologist performance.
• Including "May be appropriate" recommendations, ped-Llama reached 90% accuracy.
• ped-Llama demonstrated higher consistency (72%) across multiple runs compared to generic LLMs (44-50%).

Conclusions:

• A locally run, RAG-enabled chatbot using an open-source LLM can deliver guideline-concordant pediatric imaging recommendations with expert-level accuracy.
• This demonstrates the potential of AI-assisted decision support systems in radiology.
• Such systems offer a practical and scalable solution for improving clinical decision-making in pediatric imaging.