Multi-Query Cross-Modal Attention Fusion for Cognitive Impairment Recognition

Area of Science:

• Gerontology
• Neuroscience
• Biomedical Engineering

Background:

• Cognitive impairment affects the elderly, presenting societal and healthcare challenges.
• Early detection is vital for effective cognitive rehabilitation and management.
• Current assessment methods lack convenience and speed.

Purpose of the Study:

• To develop a novel multimodal fusion model for accurate and rapid cognitive impairment recognition.
• To leverage synchronized electroencephalography (EEG), electrocardiography (ECG), and video signals.
• To enhance early detection for improved cognitive rehabilitation strategies.

Main Methods:

• Constructed the EEV-CI dataset with synchronized EEG, ECG, and video data.
• Developed a frequency-band adaptive encoder for physiological signals.
• Utilized a multi-query cross-modal attention mechanism for signal fusion.
• Extracted facial action units and emotional states for expression analysis.

Main Results:

• The proposed model achieved 87.01% accuracy, 78.17% F1-macro, 86.88% AUC, and 57.42% MCC.
• Modality-specific experiments confirmed the contribution of each signal type.
• Demonstrated superior performance in cognitive impairment recognition.

Conclusions:

• Multimodal fusion is effective for cognitive impairment assessment.
• The developed model offers valuable clinical support for rehabilitation.
• Highlights the potential for rapid and convenient cognitive impairment detection.