Multimodal Spiking Neural Network With Generalized Distributive Law for Biosignal and Sensory Fusion.

Area of Science:

• Biomedical Engineering
• Intelligent Sensing
• Computational Neuroscience

Background:

• Multimodal signal fusion integrates diverse data (EEG, speech, imaging) for holistic analysis.
• Current methods like Transformers are computationally expensive, while STDP-based networks lack efficient topology formation.
• Efficient and biologically plausible integration of heterogeneous signals remains a challenge.

Purpose of the Study:

• To introduce a novel end-to-end framework, the Multimodal Spiking Neural Network (MSNN), for efficient and interpretable multimodal sensor fusion.
• To address the computational and biological plausibility limitations of existing fusion architectures.
• To leverage the Generalized Distributive Law (GDL) and structure-adaptive leaky integrate-and-fire (SALIF) neurons for enhanced fusion efficiency.

Main Methods:

• Developed an MSNN framework incorporating a GDL-based fusion module for integrating heterogeneous biomedical and sensory signals.
• Integrated SALIF neurons for dynamic optimization of sparse connectivity, improving fusion efficiency.
• Validated the MSNN on diverse datasets including DEAP, WESAD, MNIST, TIDIGITS, MNIST-DVS, and N-TIDIGITS.

Main Results:

• Achieved high accuracy in affective state decoding (DEAP: 92.29% valence, 91.08% arousal) and stress detection (WESAD: 99.77%).
• Demonstrated state-of-the-art performance on pattern recognition (MNIST & TIDIGITS: 99.01%) and neuromorphic datasets (MNIST-DVS & N-TIDIGITS: 99.98%).
• The MSNN framework proved versatile and effective across various multimodal sensing tasks.

Conclusions:

• The proposed MSNN offers an effective, energy-efficient solution for multimodal sensor fusion in biomedical and intelligent sensing.
• The GDL mechanism provides an efficient and interpretable approach to signal integration.
• The framework's ability to dynamically optimize sparse connectivity enhances overall fusion performance.