Medical Data Wrangling With Sequential Variational Autoencoders.

Area of Science:

• Data Science
• Machine Learning
• Biomedical Informatics

Background:

• Medical datasets frequently contain noise and missing values.
• Missing data patterns in healthcare are often non-random, occurring in bursts due to sensor issues or misaligned data collection.

Purpose of the Study:

• To propose a novel methodology, Shi-VAE, for modeling medical data with heterogeneous types and bursty missing data.
• To extend Variational Autoencoders (VAEs) for sequential data streams with missing observations.

Main Methods:

• Developed Shi-VAE, a sequential Variational Autoencoder tailored for bursty missing data in medical records.
• Compared Shi-VAE against state-of-the-art methods on Intensive Care Unit (ICU) and passive human monitoring datasets.
• Utilized cross-correlation alongside RMSE to evaluate temporal model performance.

Main Results:

• Shi-VAE demonstrated superior performance compared to existing solutions on both benchmark datasets.
• The proposed model effectively handles bursty missing data patterns common in medical scenarios.
• Shi-VAE offers improved accuracy and lower computational complexity than the state-of-the-art GP-VAE model.

Conclusions:

• Shi-VAE provides an effective solution for modeling complex missing data patterns in sequential medical records.
• The methodology advances the imputation of missing medical data, particularly when missingness occurs in bursts.
• Standard error metrics like RMSE are insufficient for temporal models; cross-correlation offers a more robust evaluation.