Partial Multiple Imputation With Variational Autoencoders: Tackling Not at Randomness in Healthcare Data

Area of Science:

• Data Science
• Machine Learning
• Biostatistics

Background:

• Missing data in healthcare research, especially from routinely collected data, can lead to biased results.
• Traditional imputation methods often fail under Missing Not At Random (MNAR) mechanisms.
• Generative models like Variational Autoencoders show promise but can produce biased imputations from single samples.

Purpose of the Study:

• To develop an advanced imputation method addressing limitations of standard Variational Autoencoders for MNAR data.
• To enhance the accuracy and reduce bias in imputing missing values in critical healthcare datasets.
• To improve the performance of machine learning models using data with missing values.

Main Methods:

• An extension of Variational Autoencoders incorporating a partial multiple imputation procedure was developed.
• The proposed method was evaluated against 8 state-of-the-art imputation strategies.
• Experiments were conducted on 34 medical datasets with simulated MNAR data at missing rates from 10% to 80%.

Main Results:

• The novel method achieved superior performance, outperforming existing strategies in 71% of the evaluated datasets.
• Significant improvements were observed, particularly at higher missing data rates (up to 80%).
• In a heart failure classification case study, the method improved the performance of 50% of the tested classifiers.

Conclusions:

• The proposed partial multiple imputation extension to Variational Autoencoders offers a robust solution for MNAR data in healthcare.
• This method significantly enhances imputation accuracy and reduces bias compared to current state-of-the-art techniques.
• The approach demonstrates potential for improving downstream machine learning tasks in clinical research.