Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening.

Area of Science:

• Artificial Intelligence in Medical Research
• Radiology Systematic Reviews
• Natural Language Processing Applications

Background:

• Systematic reviews are crucial for evidence synthesis in radiology.
• Automating title and abstract screening with large language models (LLMs) could enhance efficiency.
• Evaluating LLM performance, stability, and decision-making in this context is essential.

Purpose of the Study:

• To assess the performance, stability, and decision-making of various LLMs for radiology systematic review screening.
• To investigate the impact of prompt framing and confidence calibration on LLM accuracy.
• To evaluate LLM robustness when faced with disagreements and explore autonomous search capabilities.

Main Methods:

• Five LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 2.0 Flash, Llama 3.3 70B) were compared on two radiology systematic reviews.
• Tasks included binary/ternary classification, confidence scoring, and reclassification of disagreements.
• Autonomous PubMed retrieval was piloted using OpenAI and Gemini Deep Research tools.

Main Results:

• LLMs demonstrated high specificity but variable sensitivity, with F1 scores ranging from 0.389 to 0.854.
• Ternary classification yielded low abstention rates (<5%) and modest sensitivity gains.
• Models exhibited authority bias, favoring human labels in disagreements, though GPT-4o showed more resistance.

Conclusions:

• LLMs show potential for systematic review screening in radiology.
• Careful prompt engineering and human-in-the-loop oversight are critical for reliable LLM application.
• Further research is needed to optimize LLM integration and address decision-making biases.