SICE: an improved missing data imputation technique.

Area of Science:

• Data Science
• Machine Learning
• Statistics

Background:

• Missing data significantly degrades performance in data analytics and can lead to incorrect predictions.
• Efficiently handling missing values is crucial in the big data era due to the massive volume of data generated.
• Existing imputation methods may not be optimal for all data types, necessitating improved techniques.

Purpose of the Study:

• To propose a novel hybrid technique for missing data imputation, combining single and multiple imputation approaches.
• To extend the Multivariate Imputation by Chained Equation (MICE) algorithm for both categorical and numeric data.
• To evaluate the proposed technique against twelve existing algorithms using diverse real-world and public datasets.

Main Methods:

• Developed a hybrid imputation technique extending the Multivariate Imputation by Chained Equation (MICE) algorithm.
• Implemented two variations of the extended MICE algorithm for categorical and numeric data imputation.
• Compared the proposed method with twelve existing algorithms on sixty-five thousand real health records and three public datasets.

Main Results:

• The proposed algorithm achieved a 20% higher F-measure for binary data imputation compared to existing methods.
• The new technique demonstrated an 11% reduction in error for numeric data imputation.
• The performance improvements were achieved with comparable execution times to existing algorithms.

Conclusions:

• The proposed hybrid imputation technique offers superior performance for both binary and numeric data.
• The extended MICE algorithm effectively handles missing values in diverse datasets, including sensitive health records.
• This advancement contributes to more accurate predictions and better utilization of big data in analytics.