ASNet: Auto-Augmented Siamese Neural Network for Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Deep convolutional neural networks (CNNs) are prevalent in video-based human action recognition.
• Traditional data augmentation, like random cropping, often introduces non-informative or noisy samples, negatively impacting recognition accuracy.
• These noisy samples, containing minimal action-relevant information, can be mislabeled, reducing overall model performance.

Purpose of the Study:

• To mitigate the performance degradation caused by noisy samples in deep learning-based action recognition.
• To propose a novel framework, the Auto-augmented Siamese Neural Network (ASNet), for more effective data augmentation.
• To introduce a method for identifying and utilizing informative video patches (salient patches) for improved training.

Main Methods:

• Proposed Auto-augmented Siamese Neural Network (ASNet) framework.
• Developed a Salient Patch Agent (SPA) using reinforcement learning (Markov decision process) to extract critical video patches in a weakly supervised manner.
• Implemented gradient compensation by backpropagating salient patches and randomly cropped samples simultaneously to counteract the effects of non-informative samples.

Main Results:

• Demonstrated the effectiveness of the proposed SPA in identifying informative patches for human action recognition.
• ASNet framework showed significant improvements in action recognition performance compared to traditional methods.
• Experiments on UCF-101 and HMDB-51 datasets validated the efficacy of both SPA and ASNet.

Conclusions:

• The proposed ASNet effectively reduces the impact of noisy samples in video action recognition.
• The SPA method provides an efficient way to generate salient patches without requiring additional labels.
• This approach enhances the robustness and accuracy of deep learning models for human action recognition.