CT-Net: an interpretable CNN-Transformer fusion network for fNIRS classification

Lingxiang Liao ¹, Jingqing Lu ², Lutao Wang ¹

⁰School of Computer Science, Chengdu University of Information Technology, Chengdu, 610225, China.

Medical & Biological Engineering & Computing +

|

May 30, 2024

View abstract on PubMed

Summary

A new deep learning model, CT-Net, accurately classifies mental arithmetic tasks using functional near-infrared spectroscopy (fNIRS) brain data. This interpretable method enhances brain-computer interface capabilities.

Area Of Science

Neuroscience
Biomedical Engineering
Machine Learning

Background

Functional near-infrared spectroscopy (fNIRS) is a key optical neuroimaging technique for brain activity recognition.
Deep learning models are increasingly applied to fNIRS data classification challenges.

Purpose Of The Study

To introduce CT-Net, a novel deep learning model combining convolutional neural networks and Transformers for fNIRS-based mental arithmetic task classification.
To enhance data utilization and feature learning by exploring novel data representations, specifically a temporal-level combination of raw chromophore signals.

Main Methods

Developed CT-Net, integrating convolutional neural network and Transformer architectures.
Implemented a temporal-level combination of two raw chromophore signals for enriched feature learning.
Evaluated model performance on two open-access fNIRS datasets.

Main Results

Achieved high classification accuracies of 98.05% and 77.61% on two distinct datasets.
Demonstrated model interpretability using gradient-weighted class activation mapping, showing consistency with brain activity patterns during mental arithmetic tasks.

Conclusions

CT-Net shows significant feasibility and interpretability for decoding mental arithmetic tasks using fNIRS data.
The proposed data representation strategy effectively improves feature learning and model performance.

Keywords:

CNN Classification Gradient-weighted class activation mapping Transformer fNIRS