3D network with channel excitation and knowledge distillation for action recognition

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Modern action recognition often uses separate spatial (RGB) and temporal (optical flow) streams.
• While combining streams improves performance, optical flow computation is resource-intensive and time-consuming.
• 3D Convolutional Neural Networks (CNNs) are increasingly used for spatiotemporal feature extraction.

Purpose of the Study:

• To develop a novel method for training a 3D CNN using only RGB frames that effectively mimics the motion stream.
• To eliminate the need for optical flow calculation during the testing phase of action recognition.
• To enhance feature extraction capabilities and improve action recognition accuracy.

Main Methods:

• Proposed a Channel Excitation (CE) module for improved feature extraction in 3D networks, outperforming the SE block.
• Introduced a combined loss function using knowledge distillation and cross-entropy for training the Intensified Motion RGB Stream (IMRS).
• Trained the IMRS model using only RGB frames, integrating appearance and motion information.

Main Results:

• The CE module demonstrated superior feature extraction capabilities compared to the SE block.
• The IMRS model achieved 73.5% accuracy on the HMDB51 dataset, surpassing individual RGB (65.6%) and Flow (69.1%) streams.
• Extensive experiments validated the effectiveness and competitiveness of the proposed method against other action recognition models.

Conclusions:

• The proposed method successfully trains a 3D CNN using RGB frames to replicate motion stream features, negating the need for optical flow.
• The CE module and combined loss function significantly enhance the performance of action recognition systems.
• The IMRS approach offers a more efficient and competitive solution for behavior recognition.