Imputation by feature importance (IBFI): A methodology to envelop machine learning method for imputing missing...

Area of Science:

• Environmental Science
• Data Science
• Machine Learning

Background:

• Missing data is a common challenge in machine learning, impacting model performance and reliability.
• Existing imputation methods may struggle with complex missingness patterns, including missing not at random (MNAR).
• Accurate data imputation is crucial for analyzing environmental datasets like soil radon gas concentration (SRGC).

Purpose of the Study:

• To introduce and evaluate a novel imputation methodology: Imputation by Feature Importance (IBFI).
• To assess IBFI's effectiveness across various missing data types (MCAR, MAR, MNAR).
• To compare IBFI's performance against traditional imputation techniques using SRGC data.

Main Methods:

• IBFI utilizes feature importance derived from a base learning algorithm (XGBoost) to iteratively impute missing values.
• The method assumes that the target variable (SRGC) depends on environmental parameters (temperature, humidity).
• Performance was evaluated using metrics like RMSE, MSLE, MAPE, PB, and MSE on simulated and real-world SRGC time-series data.

Main Results:

• IBFI demonstrated efficient handling of missing data, outperforming mean, median, mode, PMM, and hot-deck methods.
• The method showed particular advantage in scenarios with multiple missing variables, often a challenge for other techniques.
• IBFI proved effective for imputing soil radon gas concentration (SRGC) data, even with complex missingness patterns.

Conclusions:

• IBFI is a robust and efficient imputation technique applicable to diverse machine learning tasks and missing data scenarios.
• The method's reliance on feature importance provides a physically meaningful approach to data imputation.
• IBFI offers a valuable advancement for environmental data analysis and other fields facing data imputation challenges.