VT-SGN:Spiking Graph Neural Network for Neuromorphic Visual-Tactile Fusion

Area of Science:

• Neuromorphic Engineering
• Artificial Intelligence
• Computational Neuroscience

Background:

• Current neuromorphic systems struggle with limited representation in training networks and insufficient cross-modal fusion for visual-tactile perception.
• Existing methods often fail to fully exploit the rich spatial and temporal information present in multimodal sensory data.

Purpose of the Study:

• To propose and validate a novel dual network, the visual-tactile spiking graph neural network (VT-SGN), addressing limitations in neuromorphic visual-tactile perception.
• To enhance the representational power and cross-modal fusion capabilities of neuromorphic systems by integrating graph and spiking neural networks.

Main Methods:

• Developed a VT-SGN framework combining graph neural networks (GNNs) and spiking neural networks (SNNs) for joint neuromorphic visual and tactile data utilization.
• Transformed spatiotemporal neuromorphic data into a taxel-based tactile graph and converted images into graph structures for GNN training and feature extraction.
• Employed SNNs for temporal expansion and backpropagation, preserving biodynamic mechanisms while improving representational power.

Main Results:

• The VT-SGN framework effectively addresses morphological variance between visual and tactile perceptions, leveraging complementary data.
• Demonstrated improved representational power by utilizing structural differences in the spatial dimension and preserving SNN mechanisms.
• Comprehensive experiments on three datasets validated the VT-SGN framework's superiority over state-of-the-art approaches in neuromorphic perceptual learning.

Conclusions:

• The proposed VT-SGN framework offers a significant advancement in neuromorphic visual-tactile perception by enhancing network representation and cross-modal fusion.
• This approach effectively integrates diverse sensory data, paving the way for more sophisticated and biologically plausible artificial perception systems.
• The VT-SGN framework shows strong potential for applications requiring robust and integrated visual-tactile understanding.