CAM-RNN: Co-Attention Model Based RNN for Video Captioning

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Video captioning integrates visual and textual data.
• Recurrent Neural Networks (RNNs) generate captions word-by-word.
• Existing methods struggle with irrelevant visual/textual information, hindering caption accuracy.

Purpose of the Study:

• To develop a more effective video captioning model.
• To address the interference caused by uncorrelated visual and textual features.
• To improve the accuracy and relevance of generated video captions.

Main Methods:

• A co-attention model based recurrent neural network (CAM-RNN) is proposed.
• The co-attention model (CAM) encodes visual and text features.
• CAM comprises visual and text attention modules with a balancing gate.

Main Results:

• The visual attention module adaptively focuses on salient video regions and frames.
• The text attention module highlights relevant previously generated words/phrases.
• The balancing gate regulates the influence of visual and text features during caption generation.

Conclusions:

• The proposed CAM-RNN effectively improves video captioning.
• Experiments on MSVD, Charades, MSR-VTT, and MPII-MD datasets validate the approach.
• The model successfully leverages correlated visual and textual features for accurate captioning.