Multi-Query Cross-Modal Attention Fusion for Cognitive Impairment Recognition

Cognitive impairment is a common health issue among the elderly, posing significant challenges to society and healthcare resources. Early recognition is a critical strategy for preventing or delaying the onset of cognitive impairment, particularly in cognitive rehabilitation. However, achieving convenient and rapid assessment of cognitive impairment remains a significant challenge. This study proposes a multi-query cross-modal attention fusion model based on synchronized multimodal signals for cognitive impairment recognition. First, we constructed a standardized cognitive impairment assessment dataset (EEV-CI) containing synchronized electroencephalography, electrocardiography, and video signals. Then, a frequency-band adaptive physiological signal encoder was designed to address the pseudo-periodic characteristics. Facial Action Units and emotional states were extracted for facial expression analysis. Finally, the multi-query cross-modal attention mechanism was employed to synchronize and fuse multimodal signals along the temporal dimension. Experimental results demonstrate that the proposed model achieved superior performance, with 87.01% accuracy, 78.17% F1-macro score, 86.88% AUC and 57.42% MCC. Furthermore, modality-specific experiments demonstrate the unique contribution of each modality to cognitive impairment recognition. These findings further validate the effectiveness of multimodal fusion in cognitive impairment assessment and offer valuable clinical support for rehabilitation.