Multimodal Deep Learning Models for Detecting Dementia From Speech and Transcripts

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computational Linguistics

Background:

• Alzheimer's dementia (AD) poses significant psychological, social, and economic burdens.
• Early diagnosis of AD is crucial but limited by current multimodal analysis methods.
• Existing approaches often concatenate or average predictions from separate speech and text models.

Purpose of the Study:

• To develop novel end-to-end trainable methods for AD detection and MMSE score prediction.
• To effectively combine speech, transcript, and visual data within a single neural network.
• To overcome limitations of existing multimodal fusion techniques in AD research.

Main Methods:

• Utilized BERT and Vision Transformer for transcript and image processing, respectively.
• Implemented a Co-Attention layer for simultaneous image and word attention.
• Introduced a Multimodal Shifting Gate and a self-attention mechanism with a gate model for inter- and intra-modal interaction.
• Processed audio into Log-Mel spectrograms and their derivatives.

Main Results:

• Achieved 90.00% accuracy in AD classification.
• Attained a Root Mean Squared Error (RMSE) of 3.61 in MMSE regression.
• Demonstrated superior performance compared to existing methods on the ADReSS Challenge dataset.
• Established a new state-of-the-art in MMSE regression.

Conclusions:

• The proposed multimodal fusion methods significantly improve AD detection and MMSE score prediction.
• End-to-end trainable models integrating diverse data modalities offer a promising direction for AD research.
• The developed approach effectively captures complex inter- and intra-modal interactions for enhanced diagnostic accuracy.