An imputation-regularized optimization algorithm for high dimensional missing data problems and beyond.

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Missing data are common in high-dimensional analyses, posing challenges for standard algorithms like expectation-maximization.
• Existing solutions are often problem-specific, lacking a generalizable approach for diverse high-dimensional missing data scenarios.

Purpose of the Study:

• To propose a novel, general algorithm for effectively handling missing data in high-dimensional statistical and machine learning problems.
• To provide a robust framework that overcomes the limitations of existing specialized methods.

Main Methods:

• A two-step iterative process involving conditional imputation of missing data and a regularized optimization step.
• Utilizing Kullback-Leibler divergence on pseudo-complete data and sparsity constraints for consistent parameter estimation in high dimensions.

Main Results:

• The proposed algorithm demonstrates consistency in parameter estimation under general conditions, even with high-dimensional data.
• Successful application illustrated across high-dimensional Gaussian graphical models, variable selection, and random-coefficient models.

Conclusions:

• The developed algorithm offers a versatile and effective solution for high-dimensional missing data problems.
• This work bridges the gap by providing a general method applicable to various complex statistical modeling tasks.