Adaptive Semantic-Enhanced Transformer for Image Captioning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Generating accurate image captions requires rich semantic information.
• Offline object detectors can introduce noise by including irrelevant semantic objects.
• Existing methods struggle to effectively filter or utilize semantic information for captioning.

Purpose of the Study:

• To propose an end-to-end adaptive semantic-enhanced transformer (AS-Transformer) model for improved image captioning.
• To enhance the extraction and utilization of relevant semantic information during the caption generation process.
• To optimize the decoding process by adaptively balancing visual and semantic attention.

Main Methods:

• Developed a constrained weakly-supervised learning (CWSL) module to reconstruct semantic object probability distributions.
• Introduced an adaptive gated mechanism (AGM) module to dynamically adjust attention between visual and semantic features.
• Integrated CWSL and AGM into an end-to-end AS-Transformer architecture for image captioning.

Main Results:

• The AS-Transformer effectively extracts and utilizes relevant semantic information, reducing noise.
• The adaptive gated mechanism successfully adjusts attention weights between visual and semantic modalities.
• Experiments on MSCOCO and Flickr30K datasets show improved caption accuracy compared to existing methods.

Conclusions:

• The proposed AS-Transformer model offers a robust mechanism for adaptive semantic enhancement in image captioning.
• The model demonstrates superior performance by effectively integrating and adapting semantic and visual information.
• This approach advances the state-of-the-art in generating more accurate and contextually relevant image captions.