Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Conventional 3D Convolutional Neural Networks (CNNs) face challenges including high computational expense, memory intensity, and overfitting.
• There is a persistent need to enhance the feature learning capabilities of 3D CNNs for complex tasks like action recognition.

Purpose of the Study:

• To introduce Spatio-Temporal Short-Term Fourier Transform (STFT) blocks as a novel alternative to traditional 3D convolutional layers.
• To significantly reduce the computational and memory complexity of 3D CNNs while improving feature representation.

Main Methods:

• Developed STFT blocks comprising non-trainable convolution layers utilizing STFT kernels for local Fourier information capture.
• Integrated trainable linear weights for learning channel correlations within the STFT blocks.
• Evaluated STFT blocks on seven diverse action recognition datasets: Something-2 v1/v2, Jester, Diving-48, Kinetics-400, UCF101, and HMDB51.

Main Results:

• STFT blocks demonstrated a substantial reduction in space-time complexity, using 3.5 to 4.5 times fewer parameters and 1.5 to 1.8 times less computation compared to state-of-the-art methods.
• The feature learning capabilities of STFT blocks surpassed those of conventional 3D convolutional layers and their variants.
• 3D CNNs incorporating STFT blocks achieved performance on par with or superior to existing state-of-the-art approaches across all tested datasets.

Conclusions:

• STFT blocks present a computationally efficient and effective solution for enhancing 3D CNNs in action recognition.
• The proposed method offers a promising direction for developing more scalable and performant deep learning models for spatio-temporal data analysis.