Explaining Multimodal Features for Screening of Cognitive Impairment Using Shapley Values | JoVE Visualize

Area of Science:

Neuroimaging
Speech analysis
Machine learning for healthcare

Background:

Current Alzheimer's disease (AD) screening often relies on single data types like medical imaging or speech.
Previous research indicates that combining multiple data sources (multimodal approaches) yields superior performance compared to single-modality models.
Understanding the contribution of each modality is crucial for developing robust and interpretable AI models for AD screening.

Purpose of the Study:

To investigate feature importance in multimodal AI models for Alzheimer's disease screening using explainable AI (XAI) techniques.
To determine the influence of brain imaging versus speech features on model decisions at both input and fusion levels for state and predictive screening.
To enhance the reliability and clinical interpretability of multimodal AI models for AD detection.

Main Methods:

Utilized explainable AI (XAI) techniques, specifically Shapley values (SHAP, GradSHAP, DeepSHAP), to analyze feature importance.
Developed and evaluated multimodal machine learning models integrating structural brain imaging and conversational speech data.
Applied both local (individual patient) and global (aggregated) analyses to identify key predictive features.

Main Results:

Multimodal models integrating brain imaging and speech data demonstrated superior performance in Alzheimer's disease screening.
Volumetric features from brain imaging were found to be more influential in classification than acoustic and linguistic features from speech.
XAI analysis provided insights into feature contributions at both the input and fusion levels for state and predictive screening.

Conclusions:

Combining structural brain imaging and speech data significantly improves Alzheimer's disease screening accuracy.
Explainable AI methods are vital for understanding and validating multimodal AI models in clinical settings.
Brain imaging features play a more critical role than speech features in the current multimodal screening models, guiding future research and clinical application.