Heterogeneity-aware multi-modal physiological signal fusion strategy based on combined contrastive learning for...

Due to the inherent non-stationary nature and significant cross-subject divergence in physiological signals, there exist prominent heterogeneity among different modalities, different channels, and different temporal patches, which will influence the fusion effectiveness of multimodal physiological signals greatly. To mitigate above heterogeneities simultaneously and enhance multimodal emotion recognition performance, a combined cross-modal contrastive learning strategy is proposed in this paper. First, a Graph Attention Network (GAT) based learnable view augmentation is introduced to simulate the variations introduced by the non-stationary nature and cross-subject divergence. Next, the temporal contrastive learning is performed between the current temporal patch and its previous temporal patches in the augmented view to mitigate the heterogeneity among different temporal patches. Then, the cross-channel contrastive learning is performed within each view and between different views to reduce both the cross-modal and cross-channel heterogeneities. Extensive experimental results under both cross-trial and cross-subject scenarios on DEAP, DREAMER, and PhyMER datasets demonstrate that: i) The proposed model outperforms state-of-the-art (SOTA) multimodal fusion models. ii) The learnable view augmentation, the temporal contrastive learning strategy, and the spatial contrastive learning strategy contribute to the performance enhancement of the proposed model. iii) The proposed model can take full advantage of the complementarities among different modalities in representing emotional states.