Rethinking Motion Representation: Residual Frames With 3D ConvNets.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• 3D Convolutional Networks (3D ConvNets) are effective for action recognition.
• Optical flow streams enhance motion representation but are computationally expensive.
• Existing methods often require high computational costs for optimal video analysis.

Purpose of the Study:

• To propose a cost-effective method for extracting motion features from videos using residual frames.
• To improve the performance of 3D ConvNets in action recognition tasks.
• To enhance motion representation and generalization capabilities in video analysis.

Main Methods:

• Utilizing residual frames as input data for 3D ConvNets, replacing traditional stacked RGB frames.
• Employing a two-path solution combining residual frame-based motion features with appearance features extracted by a 2D convolutional network.
• Evaluating the proposed method on UCF101 and HMDB51 datasets using ResNet-18-3D.

Main Results:

• Achieved significant improvements in top-1 accuracy (35.6% and 26.6% on UCF101 and HMDB51, respectively) when trained from scratch.
• Demonstrated superior motion feature extraction compared to standard RGB frames with 3D ConvNets.
• Outperformed methods using additional optical flow streams and showed better performance on unseen datasets for video retrieval tasks.

Conclusions:

• Residual frames offer a computationally efficient and effective alternative for motion representation in 3D ConvNets.
• The proposed two-path approach enhances both motion and appearance understanding for video analysis.
• The residual-input method shows strong generalization and improved performance in self-supervised learning for videos.