SAWGAN-BDCMA: A Self-Attention Wasserstein GAN and Bidirectional Cross-Modal Attention Framework for Multimodal...

Area of Science:

• Physiological signal processing
• Machine learning for affective computing
• Human-computer interaction

Background:

• Unimodal emotion recognition pipelines often fail due to limited data and poor cross-modal fusion.
• Existing methods struggle with data diversity and distributional imbalance in physiological signals.
• Accurate emotion recognition is crucial for developing intelligent human-computer interaction systems.

Purpose of the Study:

• To propose a novel framework, SAWGAN-BDCMA, to overcome limitations in unimodal emotion recognition.
• To enhance emotion recognition accuracy by synthesizing high-quality physiological data and improving cross-modal fusion.
• To provide a robust solution for data scarcity and modal imbalance in emotion recognition tasks.

Main Methods:

• Developed a Self-Attention Wasserstein Generative Adversarial Network (SAWGAN) to synthesize Electroencephalography (EEG) and Photoplethysmography (PPG) data.
• Employed a dual-branch architecture to extract discriminative spatiotemporal features within each modality.
• Implemented a Bidirectional Cross-Modal Attention (BDCMA) mechanism for adaptive, two-way fusion of multimodal physiological signals.

Main Results:

• The SAWGAN-BDCMA framework achieved 94.25% accuracy for binary and 87.93% for quaternary classification on the DEAP dataset.
• Attained 97.49% accuracy for six-class emotion recognition on the ECSMP dataset.
• Demonstrated significant accuracy improvements (0.57% to 14.01%) over state-of-the-art multimodal approaches.

Conclusions:

• The proposed SAWGAN-BDCMA framework offers a robust solution for emotion recognition challenges, including data scarcity and modal imbalance.
• This work lays a strong theoretical and technical foundation for fine-grained emotion recognition.
• The findings pave the way for more intelligent human-computer collaboration through enhanced affective computing.