Image Captioning with Bidirectional Semantic Attention-Based Guiding of Long Short-Term Memory

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Image captioning integrates computer vision and natural language processing for practical applications.
• Existing methods often directly incorporate image features into LSTM blocks, limiting dynamic adaptation.
• Refining image features based on generated text is crucial for accurate descriptions.

Purpose of the Study:

• To propose an end-to-end bidirectional semantic attention-based guiding of long short-term memory (Bag-LSTM) model for improved image captioning.
• To enhance image feature representation by dynamically leveraging text-conditional features.
• To improve the learning of long-term relations between visual and semantic information.

Main Methods:

• Utilizing a bidirectional semantic attention mechanism to guide a long short-term memory (Bag-LSTM) model.
• Fine-tuning convolution neural network parameters for text-related image feature extraction via feedback propagation.
• Employing bidirectional gLSTM as the caption generator to capture historical and future contextual information.
• Proposing variations of the Bag-LSTM model to address high-level visual-language interactions.

Main Results:

• The proposed Bag-LSTM model demonstrates effectiveness on Flickr8k and MSCOCO datasets.
• Achieved a 51.2% improvement over BRNN on the CIDEr metric.
• The model consciously refines image features based on previously generated text, leading to better descriptions.

Conclusions:

• The Bag-LSTM model offers a significant advancement in image captioning by effectively integrating semantic attention and bidirectional context.
• The proposed approach outperforms baseline algorithms, highlighting the benefits of dynamically leveraging text-conditional image features.
• Further variations of the model show promise for describing complex visual-language interactions.