Statistical testing of agreement in overlap-based performance between an AI segmentation device and a multi-expert...

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Statistical Analysis

Background:

• AI-based medical imaging devices commonly perform lesion or organ segmentation.
• Current evaluation methods rely on aggregated reference standards and metrics like Dice coefficient, which have limitations.
• Defining meaningful success criteria and establishing a gold standard for AI segmentation evaluation remain challenging.

Purpose of the Study:

• To develop a novel statistical method for evaluating AI segmentation performance.
• To assess the agreement between AI devices and multiple human experts.
• To overcome the limitations of existing evaluation methods that require a reference standard.

Main Methods:

• Proposed a paired-testing statistical method to compare AI segmentation performance against multiple human experts.
• The method evaluates AI-to-expert dissimilarity relative to expert-to-expert dissimilarity.
• Validated using statistical and image-based simulations, and applied to AI segmentation of lung images from the Lung Image Database Consortium.

Main Results:

• Statistical simulations demonstrated effective control of Type I and Type II errors.
• Image-based simulations showed acceptable performance in assessing agreement.
• The method was successfully applied to real-world medical imaging data.

Conclusions:

• Introduced a new paired-testing statistical method for AI segmentation evaluation.
• The method enables assessment of AI agreement with human experts without a reference standard.
• Provides a valuable tool for validating AI medical imaging devices.