FLGR: Fixed Length Gists Representation Learning for RNN-HMM Hybrid-Based Neuromorphic Continuous Gesture Recognition

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Conventional cameras struggle with motion blur and high energy consumption for fast-moving objects.
• Neuromorphic vision sensors offer low latency, high dynamic range, and sparse event streams by detecting pixel-level changes.

Purpose of the Study:

• To propose a novel representation learning method, Fixed Length Gists Representation (FLGR), for event-based gesture recognition.
• To address the limitations of accumulated-frame-based representations in neuromorphic sensing.
• To develop a hybrid model for continuous gesture recognition using neuromorphic data.

Main Methods:

• Developed Fixed Length Gists Representation (FLGR) using a mixture density autoencoder for event-based data.
• Implemented a Recurrent Neural Network (RNN) for FLGR sequence classification.
• Utilized a Hidden Markov Model (HMM) for localizing gestures in continuous sequences.

Main Results:

• The proposed FLGR preserves the event-driven nature of neuromorphic sensors and provides a fixed-length format suitable for sequence classifiers.
• An RNN-HMM hybrid model effectively addresses continuous gesture recognition.
• Introduced the Neuro ConGD Dataset for continuous hand gesture recognition research.

Conclusions:

• FLGR offers a new representation for event-based data, overcoming limitations of previous methods.
• The developed RNN-HMM approach enhances continuous gesture recognition accuracy.
• The Neuro ConGD Dataset will facilitate further research in high-speed, high-dynamic-range event-based sequence classification.