Multimodal insights into granger causality connectivity: Integrating physiological signals and gated eye-tracking...

Area of Science:

• Multimodal biosignal processing
• Affective computing
• Human-computer interaction

Background:

• Emotion recognition systems often struggle with accuracy and reliability.
• Integrating physiological signals like ECG and EEG offers potential for improved emotion detection.
• Eye-tracking data can provide valuable contextual information for refining biosignal analysis.

Purpose of the Study:

• To develop a novel method for enhancing emotion recognition accuracy by fusing ECG and EEG data.
• To incorporate an eye-tracking gated strategy for filtering irrelevant emotional data.
• To investigate the dynamic interactions between brain and heart activity during emotional states.

Main Methods:

• Utilized pupil diameter from eye-tracking data to filter irrelevant emotional signals.
• Estimated effective connectivity using Granger causality (GC) to capture brain-heart dynamics.
• Employed a pre-trained ResNet-18 convolutional neural network (CNN) with GC-EEG-ECG images.
• Validated the methodology on the MAHNOB-HCI database using 5-fold cross-validation.

Main Results:

• Achieved an average accuracy of 91.00% for emotion recognition.
• Obtained an Area Under the Curve (AUC) of 0.97.
• Demonstrated significant performance enhancement compared to state-of-the-art methods.

Conclusions:

• Combining ECG and EEG data with an eye-tracking strategy substantially improves emotion recognition.
• The proposed Granger causality-based approach effectively captures dynamic brain-heart interactions for emotion detection.
• This multimodal approach offers a more accurate and reliable method for affective computing.