UAT: Universal Attention Transformer for Video Captioning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Encoder-decoder models are standard for video captioning.
• Convolutional Neural Networks (CNNs) with frozen weights are commonly used for feature extraction, but limit learning and increase complexity.
• CNNs possess inductive bias (local receptive fields) that may not capture global video information.

Purpose of the Study:

• To propose a full transformer architecture for end-to-end video captioning.
• To overcome limitations of traditional CNN-based feature extraction in video captioning.
• To improve spatial and temporal information utilization in video captioning models.

Main Methods:

• Utilized a Vision Transformer (ViT) as the primary feature extraction model.
• Introduced Feature Extraction Gates (FEGs) to enrich visual features for the captioning model.
• Designed a Universal Encoder Attraction (UEA) mechanism for enhanced temporal relationship modeling.

Main Results:

• The proposed transformer model achieved competitive performance on MSRVTT and MSVD datasets.
• The model demonstrated effective video captioning using a single feature type.
• In certain scenarios, the single-feature transformer model outperformed multi-feature CNN-based approaches.

Conclusions:

• A full transformer architecture offers a viable alternative to traditional CNNs for video captioning.
• The proposed FEGs and UEA effectively enhance feature representation and temporal understanding.
• This approach simplifies the model while maintaining or improving captioning performance.