Exploiting Images for Video Recognition: Heterogeneous Feature Augmentation via Symmetric Adversarial Learning

Area of Science:

• Computer Vision
• Deep Learning
• Machine Learning

Background:

• Training deep video recognition models requires extensive labeled video data, which is costly and time-consuming to acquire.
• Large-scale video datasets demand significant computational resources, limiting accessibility for many researchers.
• Directly applying image data for video recognition suffers from domain shift and feature heterogeneity, leading to performance degradation.

Purpose of the Study:

• To propose a novel symmetric adversarial learning approach for heterogeneous image-to-video adaptation.
• To augment deep image and video features by learning domain-invariant representations in an unsupervised setting.
• To improve video recognition performance by effectively utilizing labeled images and unlabeled videos.

Main Methods:

• Developed a symmetric generative adversarial network (Sym-GAN) framework for unsupervised image-to-video adaptation.
• Learned a common image-frame feature space to bridge the gap between image and video data.
• Augmented source image features with video-specific representations and target video features with image-specific representations.

Main Results:

• The proposed Sym-GANs effectively learn domain-invariant representations, enhancing both image and video features.
• The approach successfully captures motion dynamics from video and static appearance from images.
• Experiments on UCF101 and HMDB51 datasets demonstrate superior performance compared to state-of-the-art methods.

Conclusions:

• The symmetric adversarial learning approach offers an effective solution for heterogeneous image-to-video adaptation.
• This method alleviates the need for large labeled video datasets, reducing data collection and computational costs.
• The approach shows significant potential for advancing unsupervised video recognition tasks.