MMSTG-Net: Multi-scale Spatiotemporal Graph Network with Cross-modal Contrastive Alignment for Multimodal Emotion...

Area of Science:

• Neuroscience
• Computer Science
• Psychiatry

Background:

• Current emotion recognition methods in psychiatry rely on subjective and time-limited self-reports.
• There is a need for objective, fine-grained emotion assessment tools.

Purpose of the Study:

• To develop a multimodal spatiotemporal graph network (MMSTG-Net) for precise emotion classification.
• To jointly model temporal dynamics, brain connectivity, and cross-modal data from EEG and eye-tracking.

Main Methods:

• Developed MMSTG-Net, a multimodal spatiotemporal graph network.
• Utilized electroencephalography (EEG) and eye-tracking data.
• Evaluated on SEED-VII and SEED-IV public datasets for seven-class emotion classification.

Main Results:

• Achieved state-of-the-art performance on the SEED-VII dataset (82.74% subject-dependent, 63.06% cross-subject).
• Obtained competitive results on the SEED-IV dataset (87.50% subject-dependent, 84.85% cross-subject).
• Ablation studies confirmed the effectiveness of individual network modules and fusion design.

Conclusions:

• MMSTG-Net demonstrates strong performance for emotion recognition using EEG and eye-tracking.
• The model shows potential as a research tool to aid future clinical emotion assessment.
• Further multi-center, prospective validation is required for clinical application.