fNIRS-STCT: A Novel Hybrid Spatial and Temporal CNN and Transformer Network for fNIRS Signal Classification

Functional near-infrared spectroscopy (fNIRS) is a non-invasive brain imaging method that detects neural activity by monitoring blood oxygen levels and circulation changes in the brain. Some current fNIRS classification models are simply adapted from deep learning models such as Transformer and fail to effectively leverage both temporal and spatial information from fNIRS. In this paper, we develop a hybrid deep learning model for fNIRS classification, fusing spatial convolutional neural network (CNN), a temporal CNN, and Transformer, named as fNIRS-STCT. Primarily, the architecture employs dual large convolutional kernels - one for temporal feature extraction and another for spatial feature extraction. Second, spatial and temporal CNNs are respectively integrated with Transformer by feeding CNN-extracted features as input to Transformer and then concatenating CNN and Transformer features. Third, spatial and temporal features are fused by element-wise addition. Finally, we exploit cross-entropy as the loss function, which integrates both online label smoothing and flooding strategies to avoid overfitting. The model's effectiveness is demonstrated through testing on two public fNIRS datasets. Across subject-dependent, subject-semi-dependent, and subject-independent evaluation schemes, fNIRS-STCT model consistently outperforms other compared models for all given performance metrics. Specifically, on the unilateral finger- and foot-tapping dataset, our model achieves accuracies of 83.29%, 81.73%, and 79.60% across three evaluation schemes, surpassing the second-best model by margins of 3%, 2%, and 1%, respectively. The findings underscore fNIRS-STCT's effectiveness in cross-subject mental state and brain activity interpretation.