Long Short-Term Relation Transformer With Global Gating for Video Captioning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Video captioning requires understanding complex spatial and temporal relationships between objects.
• Existing methods using self-attention or graph neural networks struggle with redundant connections, over-smoothing, and relation ambiguity.

Purpose of the Study:

• To develop a novel model for video captioning that fully leverages spatial and temporal information.
• To address limitations of previous approaches in capturing nuanced object relationships.

Main Methods:

• Constructed a Long Short-Term Graph (LSTG) to simultaneously model short-term spatial semantics and long-term transformation dependencies.
• Designed a Global Gated Graph Reasoning Module (G3RM) to control information flow and reduce relation ambiguity.
• Integrated G3RM into a Transformer architecture, replacing self-attention, to create the Long Short-Term Relation Transformer (LSRT).

Main Results:

• The LSRT model achieved superior performance on the MSVD and MSR-VTT video captioning datasets compared to state-of-the-art methods.
• Visualization results demonstrated that LSRT mitigates over-smoothing issues.
• The method enhanced the model's relational reasoning capabilities.

Conclusions:

• The proposed LSRT model effectively captures intricate object relationships in videos.
• LSRT offers a significant advancement in video captioning by improving the understanding and generation of descriptive language.