Reliability of Multimodal LLMs for Sinusitis with Polyps vs. Sinusitis Without Polyps Classification from Paranasal...

The purpose of the study is to compare the diagnostic reliability of commercially available multimodal large language models (LLMs) in distinguishing sinusitis with nasal polyps from sinusitis without polyps in representative paranasal sinus computed tomography (CT) images. This retrospective, single-center study included 175 adults (80 sinusitis with polyps, 95 sinusitis without polyps) who underwent non-contrast paranasal sinus CT. For each case, four anonymized representative slices selected through consensus were presented to three multimodal LLMs using a standardized structured prompt via a zero-shot approach in January 2025 (ChatGPT-4o) and August 2025 (ChatGPT-5 and Gemini 2.5 Pro). Model outputs were independently evaluated by a board-certified head and neck radiologist and a fourth-year radiology resident against a radiological/clinical reference standard, and performance metrics (accuracy, sensitivity, specificity, F1 score, and Cohen's kappa) were calculated; binary differences were tested using the McNemar test. ChatGPT-4o demonstrated the highest diagnostic agreement rate (accuracy 0.89; sensitivity 0.88; specificity 0.90; kappa = 0.77) and performed statistically significantly better than ChatGPT-5 and Gemini 2.5 Pro (p < 0.001 for each). ChatGPT-5 demonstrated moderate agreement (accuracy 0.67; sensitivity 0.50; specificity 0.82; kappa = 0.33). In contrast, Gemini 2.5 Pro demonstrated the lowest performance (accuracy 0.56; sensitivity 0.50; specificity 0.61; kappa = 0.11). Multimodal LLMs showed significant differences in CT-based polyp detection under the same inputs, depending on the version and vendor. Before clinical workflow integration, task-specific validation and continuous monitoring of the deployed model version are crucial.