Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing.

Area of Science:

• * Neuromorphic Engineering
• * Human-Machine Interaction
• * Signal Processing

Background:

• * Hand-gesture recognition is crucial for Human-Machine Interaction (HMI), with applications in healthcare and assistive technologies.
• * Multi-sensor fusion, combining electromyography (EMG) and visual data, enhances accuracy but faces computational cost and latency challenges.
• * Neuromorphic technologies offer low-power, real-time processing capabilities to address these limitations.

Purpose of the Study:

• * To present a fully neuromorphic sensor fusion framework for hand-gesture recognition.
• * To integrate an event-based vision sensor with neuromorphic processors (Loihi, ODIN + MorphIC).
• * To evaluate the efficiency and performance of this neuromorphic approach against traditional machine learning methods.

Main Methods:

• * Developed a sensor fusion framework using an event-based vision sensor (DVS) and neuromorphic platforms.
• * Recorded synchronized visual and electromyography (EMG) signals for five sign language gestures.
• * Designed Spiking Neural Networks (SNNs) tailored for neuromorphic chip constraints to process fused sensor data.

Main Results:

• * Achieved classification accuracy comparable to a GPU-based software baseline.
• * Neuromorphic implementation showed increased inference time (20–40%) but significantly improved energy efficiency (30×–600×).
• * Demonstrated a substantial reduction in the energy-delay product (EDP) compared to traditional methods.

Conclusions:

• * A fully neuromorphic sensor fusion approach is viable for real-world hand-gesture recognition.
• * Neuromorphic computing provides a highly efficient solution for processing complex multi-sensor data in HMI.
• * This work establishes a benchmark for advancing neuromorphic computing in practical applications.