Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning.

Area of Science:

• Neuroscience and Artificial Intelligence
• Brain-Computer Interface (BCI) Technology
• Affective Computing

Background:

• Brain-computer interface (BCI) technology is crucial for emotion recognition, especially in virtual reality (VR).
• Extensive calibration for precise emotion recognition models poses challenges for sensitive populations.
• Current methods require significant time and effort for model training and adaptation.

Purpose of the Study:

• To develop a novel approach for synthesizing electroencephalography (EEG) data using heterogeneous adversarial transfer learning (HATL).
• To reduce the extensive calibration time required for emotion recognition models in BCI applications.
• To evaluate the efficacy of different generative adversarial network (GAN) architectures within the HATL framework.

Main Methods:

• Utilized heterogeneous adversarial transfer learning (HATL) to synthesize EEG data from other signal modalities.
• Benchmarked three GAN architectures: conditional GAN (CGAN), conditional Wasserstein GAN (CWGAN), and CWGAN with gradient penalty (CWGAN-GP).
• Tested the framework on SEED-V, DEAP, and a newly collected immersive 3D dataset, GraffitiVR.

Main Results:

• Classifiers trained with CWGAN-GP-generated EEG data and non-EEG sensory data achieved high accuracy: 93% (SEED-V), 99% (DEAP), and 97% (GraffitiVR).
• The proposed approach reduced calibration time by up to 30% in the GraffitiVR dataset compared to using real EEG data.
• Demonstrated robustness and versatility across conventional and immersive VR datasets.

Conclusions:

• The HATL framework effectively synthesizes EEG data, significantly reducing calibration time for BCI emotion recognition.
• CWGAN-GP proved effective in generating high-quality synthetic EEG data for robust emotion recognition.
• The methodology offers a promising solution for enhancing BCI applications, particularly in VR environments and for sensitive user groups.