Redundancy and Attention in Convolutional LSTM for Gesture Recognition.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Convolutional Long Short-Term Memory (ConvLSTM) networks are prevalent for action/gesture recognition.
• Existing architectures integrate attention mechanisms into ConvLSTM.
• Previous work combined 3D Convolutional Neural Networks (CNN) with ConvLSTM for gesture recognition.

Purpose of the Study:

• To investigate the redundancy of spatial convolutions within ConvLSTM gates.
• To analyze the impact of attention mechanisms on ConvLSTM performance in gesture recognition.
• To develop optimized LSTM variants for spatiotemporal feature learning.

Main Methods:

• Replaced standard ConvLSTM convolutions with depthwise separable, group, and shuffle convolutions for redundancy analysis.
• Developed four ConvLSTM variants to assess attention mechanisms: gate convolution removal, input attention, and reconstructed input/output gates with channelwise attention.
• Evaluated performance on gesture recognition tasks.

Main Results:

• Spatial convolutions within ConvLSTM gates contribute minimally to spatiotemporal feature fusion.
• Attention mechanisms applied to input and output gates do not significantly improve feature fusion.
• ConvLSTM's primary strength lies in temporal fusion for learning long-term spatiotemporal features from spatial or spatiotemporal inputs.

Conclusions:

• Optimized LSTM variants embedding convolutional structures solely in the input-to-state transition are proposed.
• The findings suggest a more efficient approach to spatiotemporal feature learning in recurrent networks.
• Publicly available code facilitates further research and application of these LSTM variants.