生体信号と感覚情報の融合のための一般化分配法則を用いたマルチモーダルスパイクニューラルネットワーク

Multimodal signal fusion is a cornerstone of biomedical engineering and intelligent sensing, enabling holistic analysis of heterogeneous sources such as electroencephalography (EEG), peripheral signals, speech, and imaging data. However, integrating diverse modalities in a computationally efficient and biologically plausible manner remains a significant challenge. Transformer-based fusion architectures rely on global cross-attention to integrate multimodal information but incur high computational costs. In contrast, STDP-driven fully connected layers adopt local learning rules, which restrict their ability to autonomously form efficient sparse topologies for complex multimodal tasks. To address these issues, we propose a novel end-to-end framework-the Multimodal Spiking Neural Network (MSNN)-featuring a fusion module grounded in the Generalized Distributive Law (GDL). This principled mechanism provides an efficient and interpretable means of integrating heterogeneous biomedical and sensory signals. The MSNN further incorporates structure-adaptive leaky integrate-and-fire (SALIF) neurons, enabling dynamic optimization of sparse connectivity to enhance fusion efficiency. The proposed MSNN is validated on a range of datasets, demonstrating strong versatility: it achieves binary classification accuracies of 92.29% (valence) and 91.08% (arousal) on the DEAP dataset for affective state decoding and 99.77% on the WESAD dataset for stress detection, while delivering state-of-the-art performance on standard pattern recognition tasks (MNIST & TIDIGITS: 99.01%) and event-driven neuromorphic datasets (MNIST-DVS & N-TIDIGITS: 99.98%). These results demonstrate that MSNN offers an effective and energy-efficient solution for multimodal sensor fusion in biomedical and intelligent sensing applications.