Tensor Representations for Action Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Human action recognition in videos is challenging due to complex spatio-temporal feature interactions.
• Existing methods may not fully capture higher-order relationships crucial for fine-grained recognition.

Purpose of the Study:

• To propose novel tensor representations for compact and effective capturing of higher-order relationships in visual features for action recognition.
• To introduce sequence compatibility kernel (SCK) and dynamics compatibility kernel (DCK) for modeling spatio-temporal correlations and action dynamics.
• To explore a generalized SCK (SCK ⊕) for multi-modal inputs and local-global correlation interplay.

Main Methods:

• Developed two tensor-based feature representations: SCK and DCK.
• Introduced SCK ⊕, a generalization of SCK operating on subsequences for multi-modal data (e.g., 3D skeleton joints, deep learning scores).
• Linearized kernels for compact and efficient descriptors, utilizing higher-order tensors and Eigenvalue Power Normalization (EPN).

Main Results:

• Demonstrated the effectiveness of tensor representations on 3D skeleton action sequences, fine-grained video sequences, and standard videos.
• Showcased that the proposed tensor representations, coupled with EPN, detect higher-order feature occurrences for robust fine-grained recognition.
• Proved that the tensor-based approach with EPN acts as a Tensor Power Normalization metric for detecting specific higher-order relationships.

Conclusions:

• Novel tensor representations (SCK, DCK, SCK ⊕) offer a powerful framework for human action recognition.
• The proposed methods effectively capture complex spatio-temporal dynamics and higher-order feature relationships.
• Eigenvalue Power Normalization enhances the detection of fine-grained action patterns, outperforming simple feature counting.