Semantic-Disentangled Transformer With Noun-Verb Embedding for Compositional Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Compositional Action Recognition (CAR) faces challenges due to distribution shifts in action-object pairs.
• Existing methods often rely on external information, failing to eliminate inherent video inductive biases.
• Visually cluttered video features hinder generalization by failing to isolate objects.

Purpose of the Study:

• To propose a novel Semantic-Decoupling Transformer (DeFormer) for implicit semantic-level decoupling of object-action in high-level features.
• To enhance the generalization ability of models in Compositional Action Recognition.
• To overcome the limitations of previous CAR approaches by addressing inherent inductive biases.

Main Methods:

• DeFormer framework with two sub-modules: Objects-Motion Decoupler (OMD) and Semantic-Decoupling Constrainer (SDC).
• OMD initializes learnable tokens with annotation priors to decouple instance-level representations into appearance and motion features.
• SDC utilizes textual information for dual-contrastive association to constrain decoupled features.

Main Results:

• DeFormer demonstrates significant generalization improvements on STH-ELSE and EPIC-KITCHENS-55 datasets.
• Achieved absolute improvements of 3% to 5.4% on STH-ELSE and 4.7% to 9.2% on EPIC-KITCHENS-55 compared to baselines.
• Obtained state-of-the-art results using both ground-truth and detected annotations.

Conclusions:

• DeFormer effectively addresses the distribution shift problem in Compositional Action Recognition.
• The proposed semantic-level decoupling approach enhances model generalization.
• DeFormer represents a significant advancement in state-of-the-art CAR research.