Graph-based Spatio-Temporal Feature Learning for Neuromorphic Vision Sensing | JoVE Visualize

Area of Science:

Computer Vision
Neuromorphic Engineering
Machine Learning

Background:

Neuromorphic vision sensing (NVS) captures visual data as event spikes, offering advantages in speed, energy efficiency, and illumination robustness over active pixel sensing (APS).
Current feature representation methods for NVS lag behind APS, hindering performance in complex computer vision tasks.
The sparse and asynchronous nature of NVS data presents unique challenges for traditional processing techniques.

Purpose of the Study:

To develop a compact graph representation for NVS data to enable end-to-end learning.
To propose a novel feature learning framework for NVS that handles both appearance-based and motion-based tasks.
To introduce new large-scale NVS datasets for complex recognition tasks.

Main Methods:

A compact graph representation for NVS events is proposed, facilitating end-to-end learning with graph convolutional neural networks.
A spatial feature learning module using residual-graph convolutional neural networks (RG-CNN) is developed for appearance-based features.
A Graph2Grid block and temporal feature learning module are introduced to model temporal dependencies over extended periods.

Main Results:

The proposed framework achieves state-of-the-art performance on object classification, action recognition, and action similarity labeling tasks using NVS data.
The approach effectively preserves spatial and temporal coherence of spike events while reducing computational and memory requirements.
New datasets (ASL-DVS, UCF101-DVS, HMDB51-DVS, ASLAN-DVS) are introduced to facilitate research in complex NVS recognition tasks.

Conclusions:

The novel graph representation and learning framework significantly advance NVS capabilities for high-level computer vision.
The proposed methods offer an efficient and effective solution for processing sparse, asynchronous event data from neuromorphic sensors.
The release of new datasets will spur further development and application of NVS in real-world scenarios.