Synthesizing Images From Spatio-Temporal Representations Using Spike-Based Backpropagation.

Area of Science:

• Neuromorphic Engineering
• Artificial Intelligence
• Signal Processing

Background:

• Spiking neural networks (SNNs) are energy-efficient alternatives for neuromorphic hardware.
• Processing spatio-temporal data from spike trains is crucial for SNN applications.
• Multi-modal data fusion in spike-based environments presents unique challenges.

Purpose of the Study:

• To propose a method for synthesizing images from multiple modalities within a spike-based environment.
• To develop spiking autoencoders for converting image and audio inputs into compact spatio-temporal representations.
• To demonstrate audio-to-image synthesis using learned multi-modal representations.

Main Methods:

• Utilized spiking autoencoders to encode image and audio data into spatio-temporal representations.
• Employed a direct training algorithm with a sigmoid approximation for neuron activation to enable backpropagation.
• Benchmarked spiking autoencoders on MNIST and Fashion-MNIST datasets for reconstruction accuracy.
• Trained models to learn shared spatio-temporal representations across audio and visual modalities.
• Synthesized images from audio inputs using the learned multi-modal representations.

Main Results:

• Spiking autoencoders achieved reconstruction loss comparable to artificial neural networks (ANNs) on MNIST and Fashion-MNIST.
• Demonstrated the ability to synthesize high-fidelity images from audio inputs (TI-46 digits to MNIST).
• Achieved competitive performance against ANNs in audio-to-image synthesis tasks.

Conclusions:

• The proposed method effectively synthesizes images from audio in a spike-based environment.
• Spiking autoencoders can learn meaningful multi-modal spatio-temporal representations.
• This work advances the capabilities of neuromorphic hardware for complex sensory data processing and generation.