An Unsupervised Error Detection Methodology for Detecting Mislabels in Healthcare Analytics.

Area of Science:

• Healthcare Data Analytics
• Machine Learning in Medicine
• Clinical Informatics

Background:

• Medical datasets often suffer from imbalanced classes and errors from subjective testing and clinical variability.
• Poor data quality negatively impacts classification accuracy and reliability, hindering effective clinical decision-making.

Purpose of the Study:

• To propose an unsupervised error detection method for improving the quality of medical datasets.
• To leverage the Pattern Discovery and Disentanglement (PDD) model for identifying and removing abnormal samples.

Main Methods:

• Utilized the Pattern Discovery and Disentanglement (PDD) model to discover statistically significant association patterns in large datasets.
• Applied the algorithm to the eICU Collaborative Research Database for sepsis risk assessment.
• Clustered samples in an unsupervised manner and detected abnormal data points.

Main Results:

• The proposed algorithm outperformed K-Means clustering by 38% on full datasets and 47% on reduced datasets.
• Removing abnormal samples using the error detection approach improved the accuracy of multiple supervised classifiers by an average of 4%.

Conclusions:

• The developed algorithm offers a robust and practical solution for unsupervised clustering and error detection in healthcare data.
• Improved data quality through automated error detection can significantly enhance the reliability of clinical risk assessment and predictive modeling.