Explaining Multimodal Features for Screening of Cognitive Impairment Using Shapley Values

Many approaches for Alzheimer's disease screening focus on either medical imaging or speech as modalities. However, our previous work shows that unimodal models utilizing features from either conversational speech or structural brain imaging are outperformed by models leveraging features of both modalities simultaneously. Herein, we use XAI techniques based on Shapley values to investigate which features are most influential for the multimodal models' decisions on the fusion and the input level in both state and predictive screening. We analyze individual patients using Shapley-based methods like SHAP, GradSHAP and DeepSHAP, and derive global insights for the most important features by aggregating these local techniques. We find that the models benefit from using both modalities, although volumetric features derived from brain imaging contribute more towards the classification results than acoustic and linguistic features derived from speech in both screening types.Clinical Relevance-Applying explainability methods to multimodal machine learning models enhances reliability and robustness of trained models, can support clinicians in interpreting the deployed model's decision, and may guide selection of the most important screening modalities in the future.