Handling high-dimensional data with missing values by modern machine learning techniques.

Area of Science:

• Statistics
• Computer Science
• Bioinformatics

Background:

• High-dimensional data are prevalent in genetics, finance, and geography.
• Missing data in these datasets can introduce significant bias.
• Proper handling of missing data is crucial for accurate analysis.

Purpose of the Study:

• To explore modern machine learning techniques for handling missing data in high-dimensional settings.
• To evaluate the performance of penalized regression, tree-based methods, and deep learning (DL).
• To compare imputation-based, propensity score, and doubly robust estimators.

Main Methods:

• Utilized penalized regression, tree-based approaches (e.g., XGBoost), and deep learning (DL).
• Applied imputation-based, propensity score, and doubly robust estimation strategies.
• Conducted simulation studies and analyzed a real-world dataset.

Main Results:

• Deep learning (DL) and XGBoost demonstrated superior performance in balancing bias and variance.
• These advanced methods effectively addressed missing data challenges in high-dimensional analysis.
• Simulation studies and real-world application confirmed the benefits of DL and XGBoost.

Conclusions:

• Modern machine learning techniques, particularly DL and XGBoost, offer significant advantages for high-dimensional missing data analysis.
• These methods provide robust estimation of population means and percentiles.
• The findings have implications for various fields utilizing big data, including genetics and finance.