Evaluating Multiple Input Strategies of Large Language Models for Gallbladder Polyps on Ultrasound: Comparative Study

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Gastroenterology Diagnostics

Background:

• Gallbladder polyps are common, often benign, and detected via ultrasound, creating diagnostic workload and patient anxiety.
• Current surgical guidelines for polyps ≥1.0 cm may lead to overtreatment of nonneoplastic lesions.
• Advanced large language models (LLMs) show potential for medical image analysis, offering solutions for workload and patient consultation.

Purpose of the Study:

• To evaluate the feasibility of using LLMs (ChatGPT-4o, Claude 3.5 Sonnet) for differentiating adenomatous from nonneoplastic gallbladder polyps (≥1.0 cm).
• To compare LLM diagnostic performance against radiologists and current surgical guidelines.
• To assess different LLM input strategies: direct image analysis, feature-based text analysis, and scoring model-based text analysis.

Main Methods:

• Retrospective analysis of ultrasound images and reports for gallbladder polyps ≥1.0 cm (January 2011-January 2022).
• Evaluation of LLM performance using three strategies: LLMs-image, LLMs-text, and LLMs-model.
• Comparison of LLM diagnostic metrics (sensitivity, specificity, accuracy, unnecessary resection rate) with guideline recommendations and radiologists using a scoring system (readers/LLMs-model).

Main Results:

• The LLMs-model strategy demonstrated significantly higher intrareader agreement compared to image or text-based strategies.
• LLMs using a scoring model (LLMs-model) achieved accuracy comparable to radiologists and significantly higher than the guideline (0.35 vs 0.22).
• The LLMs-model strategy significantly reduced the unnecessary resection rate of nonneoplastic polyps (83% vs 100%) while maintaining comparable sensitivity to the guideline.

Conclusions:

• LLM capabilities in direct medical image interpretation require further enhancement.
• Text-based analysis combined with a scoring system represents the most effective current diagnostic strategy for LLMs in gallbladder polyp evaluation.
• LLMs show potential to aid in reducing overtreatment of benign gallbladder polyps.