JSSE: Joint Sequential Semantic Encoder for Zero-Shot Event Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Zero-shot learning (ZSL) excels at recognizing unseen object categories but is under-explored for dynamic events (ZSER).
• Existing ZSL research primarily focuses on static object recognition, leaving a gap in understanding dynamic event recognition.
• Transferring knowledge from seen to unseen event categories requires novel approaches beyond static object recognition.

Purpose of the Study:

• To address the challenge of Zero-Shot Event Recognition (ZSER) by leveraging semantic attributes.
• To introduce the first attribute-based gesture dataset (ZSGL) for ZSER research.
• To propose and evaluate an end-to-end model, the Joint Sequential Semantic Encoder (JSSE), for ZSER.

Main Methods:

• Developed the ZSGL dataset using Amazon Mechanical Turk, featuring 26 gesture categories and 65 attributes.
• Employed trainable recurrent networks and 3D Convolutional Neural Networks (CNNs) to extract spatio-temporal features.
• Proposed the Joint Sequential Semantic Encoder (JSSE) model for end-to-end ZSER, optimizing semantic and classification tasks.

Main Results:

• The JSSE model demonstrated superior performance compared to existing baselines across four experimental conditions (Within-category, Across-category, Closed-set, Open-Set).
• Evaluations on the ZSGL, UCF, and HMDB datasets confirmed JSSE's effectiveness in Zero-Shot Event Recognition.
• JSSE achieved favorable results, outperforming other approaches in recognizing both seen and unseen dynamic events.

Conclusions:

• The proposed JSSE model offers an effective solution for Zero-Shot Event Recognition by utilizing semantic attributes.
• The ZSGL dataset provides a valuable resource for advancing research in attribute-based ZSER.
• This work highlights the potential of semantic attribute transfer for recognizing dynamic events in a zero-shot setting.