Related Experiment Video
Updated: May 20, 2026
12:18
A Machine Learning Approach to Design an Efficient Selective Screening of Mild Cognitive Impairment
Published on: January 11, 2020
Explainable machine learning for predicting longitudinal dementia status: Establishing a leakage-free benchmark.
Mohammad Mahdi Ghiasi1,2,3, Ryan Stanley Falck1,2,3, Teresa Liu-Ambrose2,3,4
1School of Biomedical Engineering, University of British Columbia, Vancouver, British Columbia, Canada.
PLOS Digital Health
|May 18, 2026
Summary
This study establishes a leakage-free benchmark for dementia classification using longitudinal data, revealing lower realistic predictive performance after addressing methodological pitfalls. Robust models require careful data handling and feature engineering for accurate dementia prediction.
Area of Science:
- Neuroscience
- Medical Informatics
- Machine Learning
Background:
- Dementia research is hindered by methodological issues like data leakage, leading to overestimated model performance.
- Prior studies often included target information (Clinical Dementia Rating - CDR) or used inappropriate data splitting for longitudinal datasets.
- Temporal dynamics are inadequately incorporated in many dementia classification models.
Purpose of the Study:
- To establish a transparent, leakage-free benchmark for dementia status classification using longitudinal data.
- To address critical gaps in prior dementia research, including target leakage, information leakage, and inadequate temporal dynamics.
- To evaluate the impact of engineered temporal features and interpretable AI on dementia classification.
Main Methods:
- Excluded CDR from predictor sets to prevent target leakage.
- Implemented a group-aware data-splitting strategy for longitudinal data integrity.
- Engineered temporal features and employed Support Vector Classifier (SVC) and LightGBM (LGBM) for classification.
- Utilized Explainable AI (XAI) techniques (permutation importance, SHAP) for model interpretation.
Main Results:
- SVC and LGBM, using combined original and engineered features, showed superior performance.
- SVC achieved high precision (73.3%) and ROC AUC (89.1%); LGBM offered better recall (69.7%) and accuracy (69.7%).
- XAI identified key predictors: MMSE and atlas-scaling-factor dynamics for SVC; education, age, and estimated total intracranial volume for LGBM. Models performed better in females.
Conclusions:
- Eliminating data leakage significantly reduces dementia classification performance estimates, providing a more realistic benchmark.
- Transparent and interpretable dementia classification models are crucial for robust longitudinal studies.
- Further research is needed to address potential sex-based biases in dementia prediction models.
