I²Transformer: Intra- and Inter-Relation Embedding Transformer for TV Show Captioning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• TV show captioning requires integrating visual and textual information.
• Existing methods struggle to effectively utilize subtitle semantics due to information gaps.
• Subtitle data offers valuable semantic clues like actor sentiments and intentions.

Purpose of the Study:

• To propose a novel model for TV show captioning that effectively integrates video and subtitle information.
• To develop a method for organizing fragmented subtitle information and creating a unified representation.
• To enhance the understanding of relationships between visual and textual modalities.

Main Methods:

• Introduced the Intra- and Inter-relation Embedding Transformer (I²Transformer) framework.
• Developed an Intra-relation Embedding Block (IAE) using learnable graphs to capture intra-modal relationships.
• Developed an Inter-relation Embedding Block (IEE) to learn cross-attention gates and select relevant information across modalities.

Main Results:

• The I²Transformer achieved state-of-the-art performance on a public TV show captioning dataset.
• Evaluated the effectiveness of IAE and IEE on TV show retrieval and video-guided machine translation.
• Demonstrated the generalization ability of IAE and IEE blocks on related multimodal tasks.

Conclusions:

• The proposed I²Transformer effectively addresses the challenge of integrating video and subtitle data for captioning.
• The IAE and IEE blocks significantly improve multimodal representation learning.
• The model's architecture shows strong generalization capabilities across various video-and-text tasks.