ViT-ConvGAN: a hybrid model for spatiotemporal action recognition using video transformer and 3D CNN

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Action recognition in videos is crucial for applications like sports analysis and healthcare.
• Current methods face challenges in integrating global motion context with local action details, especially for complex or rapidly changing actions.

Purpose of the Study:

• To propose a novel action recognition model, ViT-ConvGAN, that enhances the understanding of both global and local motion patterns in videos.
• To improve the accuracy and reliability of video action classification, particularly for complex action sequences.

Main Methods:

• The ViT-ConvGAN model integrates Video Transformer (ViT) for long-term temporal dependencies, 3D Convolutional Neural Networks (CNN) for local motion refinement, and Conditional Generative Adversarial Networks (CGAN) for feature optimization.
• ViT captures global spatiotemporal features, 3D CNN refines local motion details, and CGAN enhances feature distinctiveness for improved classification.

Main Results:

• ViT-ConvGAN achieved 87.3% Top-1 accuracy on the UCF101 dataset and 95.2% Top-1 accuracy on the Kinetics-400 dataset.
• The model demonstrated superior performance on Kinetics-400, outperforming several state-of-the-art methods.
• Ablation studies validated the significant contributions of ViT and 3D CNN modules to feature extraction.

Conclusions:

• ViT-ConvGAN offers an effective solution for complex action recognition by synergistically combining global and local feature extraction.
• The proposed architecture provides valuable insights for designing advanced models in video action analysis.