BH-CMA: A cross-modal attention network based on brain-heart collaboration for stress cognitive state decoding

Accurate recognition of stress states under different cognitive loads is important for adaptive human computer interaction and physiological stress monitoring. However, unimodal measurements provide incomplete central or autonomic information, while indiscriminate multimodal fusion may obscure the distinct temporal and spatial organization of electroencephalogram (EEG) and heart rate variability (HRV). This study proposes BH-CMA, a compact brain and heart cross modal attention network for recognizing four stress cognitive states. Modality specific encoders preserve the frequency, temporal, and scalp spatial organization of EEG and model the temporal dynamics of HRV. EEG representations then guide cross modal temporal attention to retrieve complementary autonomic information from synchronized HRV. A linear preserving residual classifier further combines stable discriminative information from the original physiological inputs with nonlinear corrections learned by the deep pathway. BH-CMA was evaluated using data from 28 participants under four conditions defined by acute stress induction and cognitive load. Leave one subject out (LOSO) evaluation showed that BH-CMA achieved an accuracy of 58.07%, a Macro F1 of 55.54%, and a Macro AUC of 81.74%. Compared with EEG alone, the BH-CMA increased Macro F1 by 4.15% and improved 23 of 28 participants. Masking HRV also reduced Macro F1 in 21 participants, supporting the effectiveness of EEG guided cross modal attention for incorporating complementary autonomic information. The deep residual pathway significantly improved performance over the internal linear prediction and primarily refined the classification boundaries associated with cognitive load and states with high cognitive load. BH-CMA provides a compact framework for cross-subject stress cognitive state decoding using coordinated EEG and HRV information.