Multimodal Neurophysiological Transformer for Emotion Recognition.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Machine Learning

Background:

• Neural function research relies on integrating diverse data modalities like electrophysiology, imaging, and surveys.
• Existing models face challenges in handling non-aligned multimodal data and variable sampling rates.

Purpose of the Study:

• To introduce a novel neurophysiological model, the Multimodal Neurophysiological Transformer (MNT), designed to address challenges in multimodal data integration.
• To improve the modeling of long-range dependencies and inter-modal influences in neurophysiological data.

Main Methods:

• The MNT utilizes a transformer architecture with cross-attention mechanisms to encode and integrate different data modalities.
• It addresses variable sampling rates to prevent non-alignment issues between raw signals and frequency-domain features.
• Intermediary weights are assessed to understand source signal contributions to predictions.

Main Results:

• The MNT was applied to predict valence and arousal using an open-source dataset of non-aligned multimodal time-series.
• Experimental results show comparable or superior performance to existing methods in classification tasks.
• Qualitative analysis indicates the MNT can model neural influences on autonomic activity for arousal prediction.

Conclusions:

• The developed Multimodal Neurophysiological Transformer (MNT) offers a robust solution for modeling non-aligned multimodal neurophysiological data.
• The architecture demonstrates potential for various downstream applications, including Brain-Computer Interface (BCI) systems.
• MNT provides insights into how different neural signals influence predictions, enhancing model interpretability.