Seeking a Hierarchical Prototype for Multimodal Gesture Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Existing gesture recognition methods often overlook intra-class variations due to irrelevant factors.
• Multimodal gesture recognition frequently employs late-stage fusion, leading to feature redundancy and underutilization of complementary information.

Purpose of the Study:

• To propose a novel hierarchical gesture prototype framework for enhanced gesture recognition.
• To address the limitations of previous methods by highlighting gesture-relevant features and exploiting cross-modal complementarity.

Main Methods:

• A hierarchical framework with sample-level and modal-level prototypes.
• Sample-level prototype uses a memory bank to mitigate gesture-irrelevant factors (illumination, background, appearance).
• Modal-level prototype employs a Generative Adversarial Network (GAN) to extract modal-invariant features and synthesize complementary modal-specific attributes.

Main Results:

• The proposed framework effectively highlights gesture-relevant features like poses and motions.
• Experiments on three benchmark datasets show superior performance compared to state-of-the-art methods.
• Demonstrated effectiveness in mitigating distractions from gesture-irrelevant factors.

Conclusions:

• The hierarchical gesture prototype framework offers a robust approach to gesture recognition.
• The method successfully leverages feature complementarity across modalities, improving recognition accuracy.
• This work advances the field by providing a more effective way to handle intra-class variations and multimodal fusion.