Validation pipeline for machine learning algorithm assessment for multiple vendors.

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Radiology Informatics

Background:

• Machine learning (ML) algorithms are increasingly used in clinical settings, necessitating standardized methods for performance evaluation.
• Comparing independently developed ML algorithms for clinical applications requires a normalized approach to ensure reliability and accuracy.

Purpose of the Study:

• To design, build, and test an objective evaluation pipeline for assessing ML algorithms used in clinical imaging.
• To normalize performance measurement of independently developed ML algorithms using a common test dataset of chest CT examinations.

Main Methods:

• Developed a pipeline for image cohort creation, de-identification, and ground-truth labeling.
• Implemented a secure server environment for testing vendor "black box" ML algorithms on internal clinical data (100 chest CTs).
• Assessed three vendor applications for lung nodule detection, calculating recall, precision, and receiver operating characteristic curves (ROC).

Main Results:

• Vendor-1: recall 0.80, precision 0.89; Vendor-2: recall 0.75, precision 0.40; Vendor-3: recall 0.32, precision 0.39.
• Area Under the Curve (AUC) values for solid, groundglass, and part-solid nodules varied significantly across vendors.
• The developed pipeline enabled secure, internal testing of multi-vendor ML algorithms.

Conclusions:

• The study demonstrates wide variations in ML algorithm performance for lung nodule detection and classification.
• A standardized, objective ML algorithm evaluation process is crucial for clinical adoption and reliable comparison of tools.
• The developed pipeline provides a framework for consistent and secure assessment of ML algorithms in radiology.