Temporal Variance Analysis for Action Recognition.

Area of Science:

• Computational Neuroscience
• Computer Vision
• Machine Learning

Background:

• Slow Feature Analysis (SFA) models complex cells in visual cortex V1, processing appearance and motion information.
• SFA primarily uses slowly varying features, limiting its effectiveness for motion information extraction.
• Existing methods may not fully leverage temporal dynamics for robust feature representation.

Purpose of the Study:

• To introduce Temporal Variance Analysis (TVA) as a generalization of SFA for improved temporal information utilization.
• To develop a novel feature extraction method for action recognition by integrating appearance and motion cues.
• To enhance the performance of action recognition systems by better representing local features with varying temporal dynamics.

Main Methods:

• TVA learns a linear transformation matrix to project multidimensional temporal data into components with temporal variance.
• Receptive fields are learned using TVA, followed by convolution and pooling for local feature extraction, inspired by V1.
• The proposed method integrates TVA into the improved dense trajectory framework for action recognition, extracting appearance and motion features using slow and fast filters from grayscale images and optical flows.

Main Results:

• Both slow and fast features extracted by TVA are demonstrated to be valuable for low-level feature extraction.
• TVA features show superior performance compared to conventional histogram-based features on challenging datasets.
• Combining all TVA features leads to excellent results in action recognition tasks.

Conclusions:

• TVA offers a powerful generalization of SFA, effectively capturing both appearance and motion information for action recognition.
• The proposed TVA-based feature extraction method significantly outperforms existing techniques.
• Integrating diverse features extracted by TVA leads to state-of-the-art performance in action recognition.