Evaluating the state of the art in missing data imputation for clinical data

Area of Science:

• Biomedical Informatics
• Data Science
• Clinical Research

Background:

• Clinical data analysis is crucial for medical advancements but is hampered by missing values.
• Irregularly sampled and unknown data points are common in clinical time series.
• Accurate imputation of missing clinical data is essential for deriving reliable medical knowledge.

Purpose of the Study:

• To evaluate and advance the state-of-the-art in imputing missing data within clinical time series.
• To establish a benchmark for clinical time series imputation using a shared dataset with ground truth.
• To foster collaboration and innovation in addressing missing data challenges in healthcare.

Main Methods:

• A shared clinical dataset of 13 common blood laboratory tests was created.
• Ground truth for imputation performance was established by randomly removing recorded results.
• The Data Analytics Challenge on Missing data Imputation (DACMI) involved 12 international teams.

Main Results:

• Machine learning and statistical models, including LightGBM, MICE, and XGBoost, demonstrated strong imputation performance.
• Effective imputation was achieved by combining temporal and cross-sectional features.
• Overly complex models should be avoided to prevent inflated performance metrics.

Conclusions:

• Competitive models can effectively impute missing clinical time series data.
• Careful feature engineering is key to successful imputation.
• Insights from the challenge will guide future research in modeling clinical missing data.