Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Image captioning models typically use Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
• Recent advancements include self-attention mechanisms in encoder-decoder architectures.
• Current methods struggle to fully leverage image information due to a lack of semantic concepts.

Purpose of the Study:

• To develop an improved image captioning model that addresses limitations in current approaches.
• To enhance visual feature extraction by incorporating semantic information.
• To generate more accurate and diverse image captions.

Main Methods:

• Proposed a novel image-Transformer-based model incorporating image object semantic representation.
• Integrated semantic representation into the encoder's attention mechanism to enrich visual features.
• Utilized a Transformer as the decoder for the language generation module.

Main Results:

• The model demonstrated improved performance in generating accurate and diverse captions.
• Evaluated on MS-COCO and MACE datasets, showing competitive results.
• The approach effectively integrates instance-level concepts for better image comprehension.

Conclusions:

• The proposed Transformer-based model with semantic enhancement offers a promising direction for image captioning.
• Integrating semantic concepts significantly improves the quality and diversity of generated captions.
• The model achieves state-of-the-art performance, aligning with current leading approaches.