Finger Gesture Spotting from Long Sequences Based on Multi-Stream Recurrent Neural Networks

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Machine Learning

Background:

• Gesture spotting is crucial for touchless in-car interfaces.
• Automated methods must detect gestures, differentiate them from natural hand movements, and operate in real-time.
• Existing methods struggle with accuracy and online processing.

Purpose of the Study:

• To develop an effective online finger gesture spotting method for autonomous vehicles.
• To improve the accuracy and efficiency of gesture recognition in complex in-car environments.
• To address the challenge of distinguishing target gestures from natural hand movements.

Main Methods:

• A novel multi-stream recurrent neural network (RNN) architecture was proposed.
• The network merges hand and hand-location features to enhance gesture discrimination.
• The model was trained and validated on a custom finger gesture dataset captured in an autonomous car using a depth sensor.

Main Results:

• The proposed gesture spotting approach achieved significant improvements in recall (10%) and precision (15%) compared to state-of-the-art methods.
• The multi-stream RNN effectively differentiates target gestures from natural hand movements by considering spatial location.
• Integration with a 3D Convolutional Neural Network classifier further boosted overall gesture recognition performance.

Conclusions:

• The developed multi-stream RNN is a highly effective method for online finger gesture spotting in autonomous driving scenarios.
• The approach offers superior performance in distinguishing gestures from non-gesture hand movements, enhancing user interface reliability.
• This work advances the field of human-computer interaction for safer and more intuitive in-car control systems.