EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM.

Area of Science:

• Neuroscience and Biomedical Engineering
• Artificial Intelligence and Machine Learning

Background:

• Human-machine interaction and sociotechnical systems necessitate monitoring brain states for performance and safety.
• Brain signals are crucial for assistive technologies like brain-computer interfaces (BCI) and neuromodulation.
• Challenges in BCI include the complexity, non-stationarity, and low signal-to-noise ratio of brain signals, especially outside lab settings.

Purpose of the Study:

• To address data compatibility issues in hybrid fNIRS-EEG BCI systems.
• To develop a robust algorithm for integrated classification of fNIRS and EEG signals.
• To improve the accuracy and reliability of brain state detection in real-time BCI applications.

Main Methods:

• A novel recurrence plot (RP)-based time-distributed convolutional neural network (CNN) and long short-term memory (LSTM) algorithm was developed.
• Brain signals were projected into a non-linear dimension using RPs, and fed into a CNN for feature extraction without downsampling.
• LSTM was employed to learn chronological features and time-dependence for brain activity detection.

Main Results:

• The proposed RP-based CNN-LSTM model achieved average accuracies of 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI.
• Maximum accuracies reached 85.9% for fNIRS, 88.1% for EEG, and 92.4% for hybrid EEG-fNIRS BCI.
• The results demonstrate the effectiveness of the RP-based deep learning approach for BCI.

Conclusions:

• The developed RP-based deep learning algorithm successfully integrates fNIRS and EEG data for hybrid BCI.
• This novel approach overcomes traditional data compatibility challenges without information loss.
• The findings confirm the viability of this method for robust and accurate BCI systems.