Dual Position Relationship Transformer for Image Captioning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Image captioning models often lack detailed spatial understanding.
• Existing methods struggle to effectively utilize object positional relationships for improved descriptions.

Purpose of the Study:

• To develop a novel Dual Position Relationship Transformer (DPR) for enhanced image captioning.
• To improve information extraction by integrating dual position relationships (relative and absolute) into the self-attention mechanism.

Main Methods:

• Utilized Convolutional Neural Network (CNN) and Faster R-CNN for feature extraction and object detection.
• Calculated relative position (RP) and absolute position (AP) of detected objects.
• Integrated RP and AP information into the self-attention mechanism.
• Employed Long Short-Term Memory (LSTM) for text vector encoding to model sequential and temporal relationships.

Main Results:

• Achieved superior performance on the MSCOCO dataset, with Consensus-based Image Description Evaluation (CIDEr) reaching 114.6 after 30 epochs.
• Demonstrated a 2x speed improvement compared to other competitive methods.
• Ablation studies confirmed the effectiveness of the proposed dual position relationship module.

Conclusions:

• The Dual Position Relationship Transformer (DPR) effectively enhances image captioning by incorporating object positional information.
• The proposed method offers a significant improvement in both accuracy and efficiency for image description generation.