CrossModal-associated-SNN: multi-channel spiking neural networks with clustering and associative learning

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Biological sensory systems exhibit robust cross-modal integration, a capability difficult to replicate in artificial spiking neural networks (SNNs).
• Existing artificial systems struggle with efficient integration of multiple sensory inputs, limiting their robustness and generalization.
• Mayer's multi-channel learning cognitive theory provides a foundation for understanding integrated sensory processing.

Purpose of the Study:

• To develop a neuro-inspired framework, CrossModal-Associated-SNN, for synergistic integration of visual and auditory information in SNNs.
• To enhance robustness and generalization in multi-modal classification tasks using biologically plausible mechanisms.
• To explore the potential of cooperative shallow micro-networks for energy-efficient multi-modal processing.

Main Methods:

• Implemented a multi-channel, multi-network architecture for modality-specific processing.
• Utilized Spike-Timing-Dependent Plasticity (STDP) clustering and associative learning for cross-modal integration.
• Employed a cross-channel complementary strategy to refine decision-making through associative signals.

Main Results:

• The dual-channel dual-network model achieved 93% visual accuracy, 84% auditory accuracy, and 94% fusion accuracy on small-sample benchmarks.
• The dual-channel triple-network architecture reached 96% visual accuracy, 90% auditory accuracy, and a peak 97% cross-modal fusion accuracy.
• Demonstrated superior generalization and robustness in multi-modal classification compared to single-channel baselines.

Conclusions:

• CrossModal-Associated-SNN effectively mimics human sensory cognitive integration, offering a biologically plausible solution.
• Cooperative shallow micro-networks present an energy-efficient alternative for multi-modal processing tasks.
• The framework advances the development of robust, energy-efficient, multi-modal intelligent systems.