Multi-step retrieval and reasoning improves radiology question answering with large language models.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Clinical Decision Support

Background:

• Large language models (LLMs) show potential for radiology decision support.
• Conventional retrieval-augmented generation (RAG) faces limitations with complex reasoning tasks.

Purpose of the Study:

• Introduce radiology Retrieval and Reasoning (RaR), a novel multi-step retrieval framework.
• Evaluate RaR's effectiveness in improving LLM diagnostic accuracy for radiology questions.

Main Methods:

• Developed RaR, a framework with iterative summarization, retrieval, and synthesis.
• Tested 25 LLMs (0.5B-670B parameters) on 104 expert-curated and 65 board-exam radiology questions.
• Compared RaR against zero-shot prompting and conventional online RAG.

Main Results:

• RaR significantly improved mean diagnostic accuracy (75%) versus zero-shot (67%) and RAG (69%).
• Accuracy gains were most pronounced in mid-sized and smaller LLMs.
• RaR reduced hallucinations and enhanced factual grounding by providing clinically relevant evidence.

Conclusions:

• Multi-step retrieval significantly enhances diagnostic reliability in radiology LLMs.
• RaR offers a promising approach for deployable, mid-sized LLMs in clinical settings.
• Publicly available code, datasets, and RaR facilitate further research and application.