fNIRS-STCT: A Novel Hybrid Spatial and Temporal CNN and Transformer Network for fNIRS Signal Classification

Area of Science:

• Neuroscience
• Machine Learning
• Biomedical Engineering

Background:

• Functional near-infrared spectroscopy (fNIRS) is a non-invasive brain imaging technique.
• Current fNIRS classification models often fail to fully utilize temporal and spatial data.
• Deep learning models like Transformer are adapted but may not be optimal for fNIRS data.

Purpose of the Study:

• To develop a novel hybrid deep learning model for enhanced fNIRS classification.
• To effectively integrate spatial and temporal features from fNIRS data.
• To improve the accuracy and generalizability of fNIRS-based brain activity interpretation.

Main Methods:

• Developed fNIRS-STCT, a hybrid model fusing spatial CNN, temporal CNN, and Transformer.
• Employed dual large convolutional kernels for distinct spatial and temporal feature extraction.
• Integrated CNN features with Transformer and fused spatial-temporal information.
• Utilized cross-entropy loss with label smoothing and flooding to prevent overfitting.

Main Results:

• fNIRS-STCT demonstrated superior performance across subject-dependent, semi-dependent, and independent schemes.
• Achieved high accuracies on a finger- and foot-tapping dataset (e.g., 83.29% subject-dependent).
• Outperformed existing models by significant margins in all evaluation metrics.

Conclusions:

• The fNIRS-STCT model effectively leverages both spatial and temporal information for robust fNIRS classification.
• This hybrid approach offers significant improvements for brain activity and mental state interpretation using fNIRS.
• The findings highlight the potential of fNIRS-STCT for advanced neuroimaging applications.