In search of truth: evaluating concordance of AI-based anatomy segmentation models

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Computational Anatomy

Background:

• Automated anatomy segmentation using artificial intelligence (AI) is crucial for analyzing large medical imaging datasets.
• Evaluating AI segmentation models is challenging without ground truth annotations, hindering model selection.
• A standardized, practical framework is needed to assess AI segmentation performance objectively.

Purpose of the Study:

• To introduce a practical framework for evaluating AI-based anatomy segmentation models.
• To enable consistent comparison of models on datasets lacking ground truth annotations.
• To facilitate informed selection of AI models for medical image analysis.

Main Methods:

• Harmonizing segmentation results into a standard, interoperable representation for consistent labeling.
• Extending 3D Slicer for streamlined loading and comparison of harmonized segmentations.
• Utilizing interactive summary plots and OHIF Viewer for browser-based visualization and review.
• Applying the framework to evaluate 6 open-source models on 31 anatomical structures from the National Lung Screening Trial dataset.

Main Results:

• Demonstrated automated loading, structure-wise inspection, and cross-model comparison of segmentations.
• Enabled quick detection and review of problematic segmentation results.
• Showed variable performance across models, with excellent agreement for some structures (e.g., lungs) but not others (e.g., vertebrae, ribs).

Conclusions:

• Developed open-source resources including harmonization scripts, summary plots, and visualization tools.
• Provided a method to assist in segmentation model evaluation without ground truth data.
• Facilitated informed model selection for AI-based anatomy segmentation.