AutoML-Multiverse: An Instability-Aware Framework for Quantifying Analytic Variability in Alzheimer's Disease...

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Biomedical Informatics

Background:

• Machine learning (ML) models for Alzheimer's disease (AD) often produce inconsistent results, impacting their reliability and interpretability.
• Analytic variability and researcher biases contribute to the instability of ML models in AD research.
• Clinical heterogeneity and cohort differences further complicate the development of robust AD prediction models.

Purpose of the Study:

• To introduce AutoML-Multiverse, a novel framework designed to characterize how analytical choices influence ML conclusions in AD research.
• To systematically quantify analytic instability in clinical ML models for AD.
• To improve the robustness and clinical applicability of ML-based prediction models for Alzheimer's disease.

Main Methods:

• Explored a vast space of approximately 20,000 analysis pipelines using the AutoML-Multiverse framework.
• Evaluated the framework across 20 classification tasks in two independent Alzheimer's disease progression cohorts (ADNI and NACC).
• Utilized multiple data modalities including neuroimaging, clinical/cognitive data, and fluid biomarkers.

Main Results:

• AutoML-Multiverse demonstrated performance equal to or better than non-automated models across all classification tasks.
• Classification accuracy for stable vs. progressive mild cognitive impairment (MCI) was 0.68±0.06 (ADNI) and 0.63±0.08 (NACC).
• Accuracy for Alzheimer's disease (AD) vs. cognitively normal (CN) classification reached 0.97±0.01 (ADNI), with modality utility varying by task and cohort.

Conclusions:

• Analytic choices significantly impact ML model rankings and biomarker importance in AD research.
• Cross-cohort variability underscores the limitations of single-dataset studies and highlights the need for instability-aware evaluation.
• The AutoML-Multiverse framework enhances the robustness of AI-driven research by reducing analysis-driven variability and characterizing uncertainty.