Spiking Transfer Learning From RGB Image to Neuromorphic Event Stream

Area of Science:

• Neuromorphic Engineering
• Computer Vision
• Artificial Intelligence

Background:

• Event cameras and spiking neural networks (SNNs) offer low-power solutions for bio-inspired vision.
• Limited labeled event stream data hinders SNN development compared to RGB datasets.
• Existing simulation methods fail to capture high temporal resolution characteristics of event cameras.

Purpose of the Study:

• To propose a transfer learning method (R2ETL) to bridge the gap between RGB image data and event camera data.
• To enhance SNN training by leveraging knowledge from large labeled RGB image datasets.
• To address the data scarcity issue in event camera research.

Main Methods:

• Developed the R2ETL (RGB to Event Transfer Learning) framework.
• Introduced a novel encoding alignment module and a feature alignment module within R2ETL.
• Implemented a temporal centered kernel alignment (TCKA) loss function for improved transfer learning efficiency.
• Provided theoretical analysis on data requirements for deep neuromorphic models.

Main Results:

• The R2ETL framework significantly outperforms state-of-the-art SNN and artificial neural network (ANN) models trained directly on event streams.
• Experiments on N-MNIST, CIFAR10-DVS, and N-Caltech101 datasets validate the framework's effectiveness.
• Demonstrated successful transfer of knowledge from RGB images to event stream training.

Conclusions:

• The R2ETL framework effectively utilizes labeled RGB images to enhance SNN training for event camera applications.
• The proposed method addresses the challenge of limited labeled event data.
• This approach paves the way for more efficient and powerful neuromorphic vision systems.