Out-of-Distribution Detection and Radiological Data Monitoring Using Statistical Process Control.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Medical Imaging

Background:

• Machine learning (ML) models are vulnerable to performance degradation when encountering data outside their training distribution.
• Data drift is a critical concern for ML-enabled devices, potentially leading to unpredictable outcomes.
• Effective out-of-distribution (OOD) detection and data drift monitoring are essential for robust ML deployment.

Purpose of the Study:

• To introduce a new framework for OOD detection and data drift monitoring.
• To combine machine learning, geometric methods, and statistical process control (SPC).
• To evaluate the framework's efficacy in identifying OOD images and monitoring data streams over time.

Main Methods:

• Investigated feature extraction and drift quantification for OOD detection.
• Applied a framework integrating ML, geometric methods, and SPC.
• Evaluated performance on differentiating various medical imaging modalities and patient demographics.

Main Results:

• Achieved high sensitivity for OOD image detection (0.980 in CT, 0.984 in CXR, 0.854 in pediatric CXR).
• Successfully detected data stream shifts (e.g., CXR to non-CXR instantly, CT shifts within days) with a low false positive rate.
• Demonstrated the framework's modality-agnostic and model-independent nature.

Conclusions:

• The proposed framework effectively detects OOD inputs and monitors data drift in ML models.
• It offers a customizable and broadly applicable solution for various imaging modalities and ML models.
• This approach enhances the reliability and safety of ML-enabled systems in dynamic environments.