Comprehensive performance testing and external validation of an AI algorithm to detect and segment brain metastases

Artificial intelligence (AI)-based models have shown initial promise in imaging brain metastasis; however, many lack validation against advanced imaging-informed datasets, precluding external validity and limiting widespread adoption. To overcome these limitations, we performed comprehensive performance testing against reference standard metrics and externally validated an AI algorithm.

As part of its FDA-clearance process, performance testing of a previously developed U-Net-based AI model was conducted on a multi-institutional cohort with reference standard established via consensus review by 3 neuroradiologists. External validation was performed on patients imaged with dual sequences (augmented) as well as an open-access dataset (UCSF-BMSR). Evaluation metrics included sensitivity, false positive (FP) rate, positive predictive value (PPV), dice similarity coefficient (DSC), 95% Hausdorff distance, normalized surface distance, and qualitative physician assessment.

In the FDA performance testing cohort, the AI algorithm achieved a sensitivity of 90.0% (95% CI: 87.0%-94.0%), DSC of 0.86 (95% CI: 0.83-0.89), and average FP rate of 0.57 lesions. In the augmented and open-access external validation cohorts, a sensitivity of 81.4% (95% CI: 73.7%-89.1%) and 85.2% (95% CI: 83.0%-87.4%) with an average number of 0.22 and 1.19 FP lesions and DSCs of 0.70 (95% CI: 0.66-0.73) and 0.78 (95% CI: 0.77-0.78) were calculated, respectively. In the augmented external validation cohort, 38.5% of contours were rated as requiring major revisions.

This AI algorithm demonstrated promising performance via 3 unique datasets. However, given the notable rate of contour revisions, these findings support its clinical role not as an autonomous system but as a human-in-the-loop tool requiring physician oversight.