Real-World Diagnostic Performance and Clinical Utility of Artificial Intelligence-Assisted Interpretation for...

BACKGROUND. Studies of artificial intelligence (AI) for lung nodule detection on CT have primarily been conducted in investigational settings and/or focused on lung cancer screening. OBJECTIVE. The purpose of this study was to evaluate the impact of AI assistance on radiologists' diagnostic performance for detecting lung metastases on chest CT in patients with colorectal cancer (CRC) in real-world clinical practice and to assess the clinical utility of AI assistance in this setting. METHODS. This retrospective study included patients with CRC who underwent chest CT as surveillance for lung metastasis from May 2020 to December 2020 (conventional interpretation) or May 2022 to December 2022 (AI-assisted interpretation). Between the two periods, the institution implemented a commercial AI lung nodule detection system. During the second period, radiologists interpreted examinations concurrently with AI-generated reports, using clinical judgment regarding whether to report AI-detected nodules. The reference standard for metastasis incorporated pathologic and clinical follow-up criteria. Diagnostic performance (i.e., sensitivity, specificity, accuracy) and clinical utility (i.e., diagnostic yield, false-referral rate, management changes after positive reports) were compared between groups based on clinical radiology reports. Net benefit was estimated using a decision curve analysis equation. Stand-alone AI interpretation was evaluated. RESULTS. The conventional interpretation group included 647 patients (mean age, 64 ± 11 [SD] years; 394 men, 253 women; metastasis prevalence, 4.3%); the AI-assisted interpretation group included 663 patients (mean age, 63 ± 12 years; 381 men, 282 women; metastasis prevalence, 4.4%). The AI-assisted interpretation group compared with the conventional interpretation group showed higher sensitivity (72.4% vs 32.1%, respectively; p = .008), accuracy (98.5% vs 96.0%, p = .005), and frequency of management changes (55.2% vs 25.0%, p = .02), without significant difference in specificity (99.7% vs 98.9%, p = .11), diagnostic yield (3.2% vs 1.4%, p = .30), or false-referral rate (0.3% vs 1.1%, p = .10). AI-assisted interpretation had positive estimated net benefit across outcome ratios. Stand-alone AI correctly detected metastasis in 24 of 29 patients but had 381 false-positive detections in 634 patients without metastasis; only one AI false-positive was reported as positive by interpretating radiologists. CONCLUSION. AI assistance yielded increased sensitivity, accuracy, and frequency of management changes, without significantly changed specificity. False-positive AI results minimally impacted radiologists' interpretations. CLINICAL IMPACT. The findings support the clinical utility of AI assistance for CRC metastasis surveillance.