Advanced methods for missing values imputation based on similarity learning | JoVE Visualize

Area of Science:

Data Science
Machine Learning
Statistics

Background:

Real-world data analysis frequently encounters missing values, posing a significant challenge for data mining accuracy and performance.
Existing imputation methods, such as k-nearest neighbors (kNN) and hard clustering, have limitations in determining optimal parameters or handling non-separated data.
Improving record similarity is crucial for enhancing imputation accuracy.

Purpose of the Study:

To propose two novel numerical missing data imputation methods: KI (kNN and iterative imputation) and FCKI (fuzzy c-means, kNN, and iterative imputation).
To automatically estimate the optimal k value for kNN to improve record similarity.
To enhance imputation accuracy by leveraging clustering and multi-level similarity measures.

Main Methods:

KI: A hybrid approach combining kNN for neighbor selection with an automatic k value estimation and iterative imputation using global correlation.
FCKI: An enhanced hybrid method integrating fuzzy c-means clustering to identify relevant clusters, followed by kNN and iterative imputation for improved similarity and accuracy.
Performance evaluation using fifteen datasets with varying missing data ratios (MCAR, MAR, MNAR) and sizes, compared against existing methods using RMSE, NRMSE, and MAE.

Main Results:

The proposed KI and FCKI methods demonstrate superior imputation accuracy compared to existing techniques.
FCKI achieves higher accuracy by utilizing fuzzy clustering to refine neighbor selection and employing a two-level similarity approach.
Both proposed methods significantly reduce the time required for data imputation.

Conclusions:

The developed KI and FCKI imputation methods offer significant improvements in accuracy and efficiency for handling missing data in datasets.
FCKI, with its integration of fuzzy logic and enhanced similarity measures, provides a more robust solution for complex datasets.
These methods are effective across different missing data types and ratios, offering a valuable contribution to data preprocessing in data mining.