Brain-inspired multisensory integration neural network for cross-modal recognition through spatiotemporal dynamics...

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Deep Learning

Background:

• Multisensory integration in the brain enhances cognitive functions.
• Existing models often struggle with flexible integration of diverse sensory inputs.

Purpose of the Study:

• To propose a bioinspired multisensory integration neural network (MINN) for recognizing multimodal information.
• To investigate the computational principles of cross-modal recognition in neural networks.

Main Methods:

• Developed a deep learning model with parallel Convolutional Neural Networks (CNNs) for feature extraction and a Recurrent Neural Network (RNN) for integration.
• Trained the network using synthetic data for digital recognition tasks.
• Evaluated robustness against noisy and asynchronous inputs.

Main Results:

• CNNs extracted orthogonal spatial and temporal features from visual and audio inputs.
• The RNN formed a stable manifold with attractors, enabling accurate cross-modal recognition.
• MINN showed superior performance in flexible integration and recognition of multisensory information.

Conclusions:

• The MINN effectively integrates visual and audio information, mimicking biological processes.
• The model provides a framework for understanding multisensory integration mechanisms.
• This work contributes to the development of brain-inspired artificial intelligence.