Video Captioning with Object-Aware Spatio-Temporal Correlation and Aggregation.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Accurate video captioning requires comprehensive understanding of video content, including object dynamics and spatial relationships.
• Existing methods often struggle to capture intricate spatio-temporal object interactions within and across video frames.

Purpose of the Study:

• To propose an object-aware spatio-temporal graph (OSTG) approach for improved video captioning.
• To effectively model inter-frame object dynamics and intra-frame spatial relationships for richer video understanding.

Main Methods:

• Constructing bidirectional temporal graphs for object alignment across frames.
• Building spatial relation graphs to learn intra-frame object interactions based on location and semantics.
• Employing trainable Vector of Locally Aggregated Descriptors (VLAD) for object-aware feature aggregation.
• Utilizing a hierarchical attention mechanism to weigh object instance contributions.

Main Results:

• The OSTG approach demonstrated state-of-the-art performance on MSR-VTT and MSVD datasets.
• Achieved superior results in BLEU@4, METEOR, and CIDEr metrics for video captioning accuracy.

Conclusions:

• The proposed object-aware spatio-temporal graph method significantly advances video captioning capabilities.
• Effective modeling of object relationships and feature aggregation leads to more accurate and comprehensive video descriptions.