From code sharing to sharing of implementations: Advancing reproducible AI development for medical imaging through...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Reproducibility in AI Research
• Federated Testing Methodologies

Background:

• The AI research community faces a significant reproducibility crisis, hindering scientific progress.
• Current solutions like code sharing are insufficient for ensuring consistent AI model performance.
• Federated testing offers a novel approach to evaluate AI models across diverse environments.

Purpose of the Study:

• To systematically evaluate sources of discrepancy in shared AI model execution for medical imaging.
• To assess the effectiveness of federated testing in ensuring reproducible AI research.
• To identify key factors influencing AI model performance in distributed settings.

Main Methods:

• Exploratory study distributing identical AI model code to multiple independent centers.
• Monitoring model execution across varied runtime environments and computational resources (GPU vs. CPU).
• Comparative evaluation of AI-driven positron emission tomography (PET) imaging segmentation across centers with distinct pre-/post-processing steps.

Main Results:

• Open-source code sharing alone does not guarantee reproducible AI results due to execution and environment variations.
• Variability in data preparation and pre-/post-processing techniques significantly impacts PET imaging segmentation accuracy.
• Computational resources (GPU/CPU) and containerization (Docker) had no discernible effect on reproducibility.

Conclusions:

• Reproducibility in federated testing necessitates comprehensive pipeline sharing, including pre- and post-processing steps.
• Standardizing protocols and sharing complete AI model pipelines are essential for robust and reliable results.
• Cloud-based platforms can automate processes to streamline AI model testing and enhance reproducibility.