Stacked Denoising Tensor Auto-Encoder for Action Recognition With Spatiotemporal Corruptions

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Action video recognition is hindered by spatial and temporal data corruption from sources like moving cameras.
• Simultaneously addressing both spatial and temporal corruptions in videos is a significant challenge.
• Existing models struggle to effectively handle arbitrary information loss in spatiotemporal data.

Purpose of the Study:

• To propose a novel model, the coupled stacked denoising tensor auto-encoder (CSDTAE), for robust action video recognition.
• To develop a unified approach that tackles both spatial and temporal corruptions concurrently.
• To enhance the performance of video recognition models in the presence of significant data loss.

Main Methods:

• The CSDTAE model utilizes a divide-and-conquer strategy, integrating spatial and temporal processing schemes.
• Each scheme employs a stacked denoising tensor auto-encoder (SDTAE), built upon denoising tensor auto-encoder (DTAE) blocks.
• Tensor unfolding and canonical correlation analysis are used to preserve spatiotemporal structure and couple corruption schemes.

Main Results:

• The CSDTAE model demonstrates significant effectiveness in recognizing action videos with substantial spatial and temporal corruptions.
• Experiments on three action datasets confirm the model's superior performance compared to existing methods.
• The approach successfully handles data corruption by treating it as noise in spatial or temporal directions.

Conclusions:

• The proposed CSDTAE model offers an effective solution for action video recognition under challenging spatiotemporal corruption conditions.
• This research advances the field by providing a unified framework for handling complex video data degradation.
• The findings highlight the potential of tensor-based methods for robust video analysis.