SFT-HN: a novel spatial-frequency-temporal hybrid network for EEG-based emotion recognition

Electroencephalograph (EEG) emotion recognition is a key task in the brain-computer interface(BCI) field. A mounting quantity of studies have shown that deep learning methods for emotion recognition exhibit superior performance compared to traditional techniques. However, it is still challenging to fuse the EEG's Spatial, Frequency and Temporal information, as well as how to make full use of discriminative local patterns among the features for different emotions. To address these issues, a novel hybrid model called Spatial-Frequency-Temporal Hybrid Network(SFT-HN) is proposed. This model includes three Spatial Frequency Residual Modules (SFRM) and an attention-based Bidirectional Long Short-Term Memory (ATBI-LSTM). The former module extracts spatial-frequency features, while the latter learns temporal contexts. SFT-HN is trained to seize the complementarity among the spatial-frequency-temporal information and adaptively explore discriminative local patterns. Specifically, 4D representations are created from raw EEG signals to preserve spatial, frequency, and temporal information. The SFRM module then adopts split-convert-merge techniques, residual and attention mechanisms to enhance its spatial-frequency feature extraction ability for each input 4D representation tensor time slice. Moreover, an attention-enhanced mechanism is incorporated into a bidirectional LSTM module to capture the crucial temporal dependencies among the extracted features, thereby enhancing the discriminative power of the EEG features. The proposed method attains average accuracies of 97.61% and 97.57% for arousal-based and valence-based classification on the DEAP dataset, respectively. On SEED dataset, the method achieves average accuracy of 97.44%. Furthermore, we validate the robust generalization of our proposed model on a novel dataset, FACED, achieving an average accuracy of 96.24%. The model code is available at: https://github.com/AllGGI/SFT-HN-model.