Spatio-Temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Few-Shot Action Recognition (FSAR) faces challenges due to limited labeled data for novel action categories.
• Existing methods rely on coarse action names, providing insufficient background knowledge for visual feature learning.
• There is a need for richer contextual information to capture complex spatial and temporal action dynamics.

Purpose of the Study:

• To propose DiST, a Decomposition-incorporation framework for enhancing FSAR.
• To utilize decoupled spatial and temporal knowledge from large language models for learning expressive prototypes.
• To improve the understanding of novel spatial and temporal concepts in action recognition.

Main Methods:

• Decomposition of action names into diverse spatio-temporal attribute descriptions.
• Incorporation of spatial and temporal knowledge using Spatial/Temporal Knowledge Compensators (SKC/TKC).
• Learning of object-level and frame-level prototypes guided by commonsense knowledge.

Main Results:

• DiST effectively learns multi-granularity prototypes by integrating commonsense knowledge.
• SKC aggregates patch tokens using spatial knowledge for object-level understanding.
• TKC models inter-frame temporal relations using temporal attributes for frame-level understanding.
• DiST achieves state-of-the-art performance on five standard FSAR datasets.

Conclusions:

• The DiST framework offers a novel approach to FSAR by leveraging decomposed knowledge from LLMs.
• The learned prototypes provide transparency in capturing fine-grained spatial details and diverse temporal patterns.
• DiST significantly advances the capabilities of action recognition systems with limited training examples.