Visual spatial relationship sensitive transformer for image captioning.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Image captioning integrates computer vision and NLP to describe visual content.
• Current methods struggle with spatial misalignment when combining grid and region features, hindering visual-semantic coherence.
• Accurate relational and contextual information is crucial for generating meaningful image descriptions.

Purpose of the Study:

• To propose a novel approach for enhancing image captioning by addressing spatial misalignment issues.
• To develop a unified visual-spatial representation by aligning multi-level features.
• To improve the model's ability to capture both global relational cues and local spatial details.

Main Methods:

• Introduced Spatial Alignment Positional Encoder (SAPE) to encode aligned grid-level and region-level features.
• Developed Group Normalization Multi-head Attention (GNMA) for global relational cues and Convolution-based Feature Enhancement Attention (CFEA) for local details.
• Proposed Learnable Adaptive Positional Encoder (LAPE) to preserve positional signals during deep training.
• Integrated these components into a transformer-based architecture named Visual-Spatial Relationship Sensitive Transformer (VRST).

Main Results:

• Achieved a CIDEr score of 141.9 on the MSCOCO dataset (Karpathy test split).
• Attained a CIDEr score of 138.2 on the MSCOCO official evaluation server.
• Demonstrated superior performance compared to several strong baseline models.

Conclusions:

• The proposed VRST effectively addresses spatial misalignment in image captioning.
• The integration of SAPE, GNMA, CFEA, and LAPE significantly enhances visual-semantic relationship modeling.
• The approach achieves state-of-the-art results on the MSCOCO dataset, showcasing its effectiveness.