A multimodal functional structure-based graph neural network for fatigue detection.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Fatigue detection is crucial, with multimodal fusion showing promise.
• Existing methods often neglect functional connectivity between signals.
• Integrating electroencephalogram (EEG) and electrocardiogram (ECG) offers a richer data source.

Purpose of the Study:

• To propose a novel multimodal fatigue classification framework integrating EEG and ECG signals.
• To address the limitation of overlooking functional connectivity in multimodal fatigue detection.
• To enhance fatigue classification accuracy by capturing inter-signal interactions.

Main Methods:

• Extracted differential entropy (DE) from EEG and heart rate variability (HRV) from ECG as dual input streams.
• Constructed cross-modal interaction graphs using correlation coefficients, Laplacian eigenvalues, and singular value decomposition (SVD).
• Employed an intra- and inter-channel separable convolution module within a graph neural network for deep pattern extraction and adaptive channel weighting.

Main Results:

• The framework effectively captured multimodal features indicative of fatigue states.
• Experiments were conducted using 64-channel (63 EEG + 1 ECG) and 17-channel (16 EEG + 1 ECG) configurations.
• Both binary and four-class fatigue classification tasks were performed, demonstrating the framework's efficacy.

Conclusions:

• The proposed framework successfully integrates EEG and ECG signals for fatigue detection.
• It effectively captures functional connectivity and deep interaction patterns between multimodal signals.
• This provides a new and effective solution for multimodal fatigue classification.