Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Gesture recognition is crucial for intuitive human-computer interaction.
• Existing multi-modal methods struggle with efficient spatio-temporal fusion.
• Manual network design limits performance in multi-modal gesture recognition.

Purpose of the Study:

• To propose the first neural architecture search (NAS)-based method for RGB-D gesture recognition.
• To enhance temporal representation and optimize multi-modal fusion for improved accuracy.
• To explore synergies between RGB and depth data for robust gesture recognition.

Main Methods:

• Developed a novel NAS framework tailored for RGB-D gesture recognition.
• Introduced 3D Central Difference Convolution (3D-CDC) for richer temporal feature extraction.
• Designed optimized backbones with multi-sampling-rate branches and cross-modal lateral connections.

Main Results:

• Achieved state-of-the-art performance on IsoGD, NvGesture, and EgoGesture datasets.
• Demonstrated superior results in both single-modality and multi-modality settings.
• The proposed multi-modal multi-rate network effectively captures spatio-temporal dynamics.

Conclusions:

• The NAS-based approach with 3D-CDC significantly advances RGB-D gesture recognition.
• Effective integration of spatio-temporal modalities is key to high-performance gesture recognition.
• The method offers a new perspective on leveraging multi-modal data for gesture understanding.