Guideline-concordant decision support in abdominal radiology: An expert-based evaluation of Gemini

Large language models may help radiologists apply increasingly complex imaging guidelines, but their safety and clinical usefulness as decision-support tools in abdominal radiology remain uncertain. We evaluated the usefulness, potential harm, guideline concordance, and triage performance of Gemini across different levels of case difficulty.

In this expert-based cross-sectional study, 200 synthetic, guideline-based abdominal radiology cases were created by consensus across four domains: adrenal, renal, liver, and pancreatic lesions (50 cases each). Case difficulty was prospectively assigned as level 1, 2, or 3 before LLM querying. Using text-only case descriptions, Gemini 1.5 Flash generated an impression, recommendation, rationale, and final category for each case. Five abdominal radiologists independently rated usefulness (1-5) and harm (0-3). Two abdominal radiologists independently assessed impression concordance, final-category concordance, and triage appropriateness relative to the reference standard.

Median usefulness was 5 (IQR, 1), and median harm was 0 (IQR, 0). Overall impression concordance was 85.0 % (170/200), and final-category concordance was 77.5 % (155/200). Greater case difficulty was associated with lower usefulness (p < 0.001), higher harm (p < 0.001), lower concordance (both p < 0.001), and worse triage performance. Triage distributions differed by case difficulty (p = 0.012), with appropriate triage declining from 85.7 % in level 1 cases to 66.2 % in level 3 cases.

Gemini showed high perceived usefulness, low overall harm, and generally strong guideline concordance in abdominal radiology. However, declining performance with greater case difficulty supports its use as assistive decision support rather than autonomous management in complex or high-risk scenarios.