An image caption model based on attention mechanism and deep reinforcement learning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Image captioning technology converts visual features into semantic information for tasks like classification and retrieval.
• Deep neural networks and encoder-decoder architectures have advanced image captioning, but challenges like visual information loss persist.

Purpose of the Study:

• To propose a novel method addressing visual information loss and non-dynamic adjustment in image captioning.
• To enhance the detail and semantic richness of generated image descriptions.

Main Methods:

• A guided decoding network connecting encoding and decoding for dynamic adjustment.
• Utilizing Dense Convolutional Network (DenseNet) and Multiple Instance Learning (MIL) for image encoding.
• Employing Nested Long Short-Term Memory (NLSTM) as the decoder.
• Incorporating an attention mechanism and a double-layer decoding structure.
• Training the model using Deep Reinforcement Learning (DRL) to optimize evaluation metrics.

Main Results:

• The proposed model demonstrates improved performance compared to existing methods.
• Enhanced capability in extracting and parsing image information.
• Generation of more detailed descriptions with richer semantic information.
• Successful training and testing on MS COCO and Flickr 30k datasets.

Conclusions:

• The novel guided decoding network effectively addresses visual information loss and improves image captioning.
• The integration of DenseNet, MIL, NLSTM, attention, and DRL leads to superior performance.
• The model shows significant improvements in BLEU, METEOR, and CIDEr evaluation metrics.