Robustness of Healthcare ML Under Data Quality Degradation: A Dimension-Wise Analysis on MIMIC-IV

Area of Science:

• Healthcare Machine Learning
• Data Quality in AI
• Clinical Informatics

Background:

• Machine learning models are increasingly used in healthcare.
• The performance of these models is sensitive to data quality.
• Understanding data quality's impact is vital for reliable AI deployment.

Purpose of the Study:

• To assess the robustness of healthcare machine learning models against controlled data quality degradations.
• To identify which data quality dimensions most impact model performance.
• To evaluate model performance when degradations occur during training versus inference.

Main Methods:

• Utilized the MIMIC-IV database with 1,500-7,000 patients.
• Applied controlled data quality degradations across five dimensions (completeness, coherence, validity, precision, uniqueness).
• Evaluated model performance on training and test data subjected to these degradations.

Main Results:

• Model performance decreased as data quality degradation increased.
• The most significant performance drops occurred when degradations were applied at inference time.
• Completeness, coherence, and validity were the most detrimental data quality dimensions.
• Models showed some adaptation to training-time imperfections but remained vulnerable to novel inference-time degradations.

Conclusions:

• Healthcare AI model robustness is significantly challenged by data quality issues, particularly at inference.
• Ensuring data completeness and coherence during inference is critical for trustworthy clinical AI systems.
• Model adaptation to training data imperfections does not guarantee resilience to deployment-time data quality variations.