Sequential Video VLAD: Training the Aggregation Locally and Temporally

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Effective video analysis requires integrating spatial and temporal information.
• Recurrent Convolution Networks (RCNs) offer a native framework for learning spatio-temporal video features.

Purpose of the Study:

• To develop a novel framework, SeqVLAD, integrating trainable Vector of Locally Aggregated Descriptors (VLAD) encoding with RCNs.
• To improve RCNs by proposing a shared Gated Recurrent Unit (GRU) architecture (SGRU-RCN) for reduced parameters and overfitting.
• To evaluate the proposed SeqVLAD and SGRU-RCN methods on video captioning and action recognition tasks.

Main Methods:

• Sequential convolutional feature maps from video frames are processed by RCNs to learn spatio-temporal assignments.
• A novel sequential VLAD layer (SeqVLAD) aggregates spatial and motion information.
• An improved GRU-RCN architecture (SGRU-RCN) is proposed with shared input-to-hidden parameters.

Main Results:

• SeqVLAD effectively combines VLAD encoding with RCNs for spatio-temporal feature learning.
• The SGRU-RCN architecture demonstrates fewer parameters and reduced overfitting compared to standard RCNs.
• Experimental results on benchmark datasets (MSVD, MVAD, UCF101, HMDB51) confirm the method's effectiveness and strong performance.

Conclusions:

• The proposed SeqVLAD framework, particularly with the SGRU-RCN architecture, significantly enhances video analysis capabilities.
• This approach offers a robust method for capturing complex spatio-temporal dynamics in videos.
• The findings suggest broad applicability in video understanding tasks like captioning and action recognition.