A balanced multimodal decoding framework of EEG and fNIRS for motor imagery

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) exhibit complementary advantages in temporal and spatial resolution for brain activity monitoring, and their integration has the potential to improve motor imagery (MI) decoding performance. However, in EEG-fNIRS multimodal MI decoding, modality heterogeneity and differences in signal-to-noise ratio and convergence speed often cause joint training to bias toward the modality that is easier to learn, leading to modality dominance, degraded fused representations, and reduced generalization. To address this issue, we propose a dynamic re-initialization framework for EEG-fNIRS multimodal decoding, which achieves cross-modal balanced learning through a diagnosis-adjustment-re-initialization mechanism. The proposed method uses the separability difference of unimodal features between the training and validation sets as a diagnostic signal, and integrates network hierarchical priors with modality-specific gradient statistics to derive adjustment factors. Guided by these signals, the framework periodically performs soft re-initialization of the EEG and fNIRS encoder parameters during training, promoting re-learning of weaker modalities, suppressing single-modality dominance, and preserving the stability of converged representations. As a result, multimodal imbalance is alleviated and the discriminative capability of fused representations is enhanced. Experiments on a publicly available EEG-fNIRS motor imagery dataset demonstrate that the proposed method achieves an average classification accuracy of 92.42 ± 4.54%, significantly outperforming both unimodal and conventional fusion baselines and showing strong cross-subject consistency.