JSE: Joint Semantic Encoder for zero-shot gesture learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Zero-shot learning (ZSL) enables recognition of unseen categories using descriptions.
• Gesture recognition (ZSGL) is an underexplored area within ZSL, lacking research in feature selection.
• Deep learning often reduces the need for feature engineering, but domain knowledge is crucial for scarce data.

Purpose of the Study:

• To investigate the impact of velocity, heuristical, and latent features on ZSGL performance.
• To propose a novel bilinear auto-encoder approach, the Joint Semantic Encoder (JSE), for ZSGL.
• To evaluate JSE's effectiveness against existing ZSL methods.

Main Methods:

• Developed the Joint Semantic Encoder (JSE), a bilinear auto-encoder.
• JSE minimizes reconstruction, semantic, and classification losses simultaneously.
• Compared JSE with existing ZSL methods using attribute-based and across-category scenarios.

Main Results:

• JSE outperformed other approaches by 5% (p<0.01) in attribute-based classification, regardless of feature type.
• When trained with heuristical features in an across-category setting, JSE showed significant performance gains (p<0.01).

Conclusions:

• The proposed JSE model demonstrates superior performance in zero-shot gesture recognition.
• Feature extraction techniques significantly influence ZSGL performance, with heuristical features showing promise.
• JSE offers a robust solution for ZSGL, particularly in data-scarce scenarios.