Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening

To evaluate the performance, stability, and decision-making behavior of large language models (LLMs) for title and abstract screening for radiology systematic reviews, with attention to prompt framing, confidence calibration, and model robustness under disagreement.

We compared five LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 2.0 Flash, Llama 3.3 70B) on two imaging-focused systematic reviews (n = 5438 and n = 267 abstracts) using binary and ternary classification tasks, confidence scoring, and reclassification of true and synthetic disagreements. Disagreements were framed as either "LLM vs human" or "human vs human." We also piloted autonomous PubMed retrieval using OpenAI and Gemini Deep Research tools.

LLMs achieved high specificity and variable sensitivity across reviews and tasks, with F1 scores ranging from 0.389 to 0.854. Ternary classification showed low abstention rates (<5%) and modest sensitivity gains. Confidence scores were significantly higher for correct predictions. In disagreement tasks, models more often selected the human label when disagreements were framed as "LLM vs human," consistent with authority bias. GPT-4o showed greater resistance to this effect, while others were more prone to defer to perceived human input. In the autonomous search task, OpenAI achieved moderate recall and high precision; Gemini's recall was poor but precision remained high.

LLMs hold promise for systematic review screening tasks but require careful prompt design and circumspect human-in-the-loop oversight to ensure robust performance.