Empirical data drift detection experiments on real-world medical imaging data.

Area of Science:

• Medical Artificial Intelligence
• Data Science
• Clinical Informatics

Background:

• Clinical artificial intelligence (AI) models require ongoing monitoring for performance degradation.
• Monitoring input data for systemic changes, known as data drift, is less common but vital for AI reliability.
• Data drift detection is particularly important when real-time performance evaluation is impractical or relies on automated labels.

Purpose of the Study:

• To evaluate the efficacy of three data drift detection methods in real-world medical imaging datasets.
• To assess the ability of these methods to detect both naturally occurring and synthetically induced data drift.
• To underscore the necessity of data drift monitoring in clinical AI deployments.

Main Methods:

• Empirical experiments were conducted using real-world medical imaging data.
• Three distinct data drift detection techniques were applied and assessed.
• Data drift was induced both naturally (e.g., emergence of COVID-19 in X-rays) and synthetically.

Main Results:

• Performance monitoring alone is an inadequate proxy for detecting data drift in clinical AI.
• The effectiveness of data drift detection is significantly influenced by sample size and patient features.
• Existing data drift detection methods show variable performance depending on the type and cause of drift.

Conclusions:

• Data drift detection is a critical, yet often overlooked, component of robust clinical AI deployment.
• Future research should address practical application challenges and knowledge gaps in data drift detection methods.
• Integrating data drift monitoring alongside performance metrics enhances the reliability and safety of AI in healthcare.