A novel multi-modal fusion framework based on simultaneous EEG-fNIRS for the classification of working memory loads

The classification of working memory load (WML) has the potential to enhance individual performance or the resilience of systems. Multi-modal fusion has been demonstrated to overcome the constraints inherent of a single modality. The study proposed a novel multi-modal fusion framework based on simultaneous electroencephalography (EEG)-functional-near-infrared-spectroscopy (fNIRS) for the classification of WML. The proposed fusion framework was comprised of three sections: the feature-extraction, the feature-level fusion, and the decision-level fusion. In the feature-extraction process, the impact of the envelope extraction, the normalization, the neural synchronization (NS), and the bilinear common spatial pattern (BCSP) on the accuracy of classification was respectively investigated through a series of ablation experiments. Experimental results demonstrated that the BCSP and the envelope exhibited a more significant impact on improving classification accuracy. In the feature-level fusion, the accuracy of the element-wise addition was found to be significantly superior to that of concatenation, element-wise multiplication, and without fusion. In the decision-level fusion, the accuracy of the majority voting (MV) was significantly superior to that of the dempster-shafer-theory (DST). To validate the superiority of the proposed fusion framework, a comparison analysis was undertaken in an open EEG-fNIRS dataset. We observed that the accuracy of the proposed fusion framework was superior to that of other multi-modal fusion methods.