A multimodal functional structure-based graph neural network for fatigue detection

Fatigue detection remains a critical research focus in the field. Recent studies have attempted to enhance detection performance through multimodal information fusion, yet they largely overlook the impact of functional connectivity among multimodal signals. To address this limitation, we propose a novel multimodal fatigue classification framework integrating electroencephalogram (EEG) and electrocardiogram (ECG) signals. The framework employs differential entropy (DE) features extracted from filtered EEG signals and heart rate variability (HRV) features derived from ECG signals as dual input streams, capturing distinct internal-external interaction patterns. Specifically, we first construct cross-modal interaction graphs by calculating correlation coefficient matrices between DE and HRV features, utilizing Laplacian eigenvalues and singular value decomposition (SVD). An innovative intra- and inter-channel separable convolution module is designed to extract deep interaction patterns through parallel convolution operations within and across signal channels. The graph neural network dynamically generates frequency-band-channel correlation matrices and adaptively assigns channel weights through learnable parameters. To evaluate channel configuration effects, we conducted experiments with two electrode configurations: 64-channel (63 EEG + 1 ECG) and 17-channel (16 EEG + 1 ECG), performing both binary and four-class classification. The experimental results show that the framework is able to effectively capture multimodal features in fatigue state and provides a new solution for fatigue classification.