Data imputation through the identification of local anomalies

Area of Science:

• Data Science
• Machine Learning
• Statistical Modeling

Background:

• Localized data corruptions from noise sources, like occluders, pose challenges in data analysis.
• Existing methods may struggle with efficient detection and accurate imputation of such corruptions.

Purpose of the Study:

• To introduce a comprehensive, model-free statistical framework for treating localized data corruptions.
• To develop a novel algorithm for detecting and localizing corruptions and a maximum a posteriori estimator for data imputation.

Main Methods:

• A binary partitioning tree is used to split data instances, with iterative anomaly detection based on clean reference data statistics.
• A novel distance measure based on ranked deviations is proposed for superior corruption separation.
• Analytical derivation shows the false alarm rate is independent of data and requires no parameter tuning.

Main Results:

• The proposed algorithm successfully detects, localizes, and imputes corrupted data with high accuracy.
• The novel distance measure outperforms Euclidean distance in separating corruptions.
• The framework demonstrates significant improvements in machine learning classification tasks.

Conclusions:

• The developed framework offers a robust and efficient solution for handling localized data corruptions.
• The approach shows strong corruption separation capabilities and outperforms traditional methods.
• The algorithms are robust across various training conditions, enhancing their practical applicability.