Fine-Grained Image Captioning by Ranking Diffusion Transformer.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• CLIP visual feature-based image captioning models have advanced rapidly.
• Existing models face challenges in generating descriptive and discriminative captions due to insufficient fine-grained visual cue exploitation and complex vision-language alignment modeling.

Purpose of the Study:

• To address limitations in current image captioning models.
• To propose a novel approach for fine-grained image captioning that enhances descriptive and discriminative capabilities.

Main Methods:

• Introduced the Ranking Diffusion Transformer (RDT) model.
• Integrated a Ranking Visual Encoder (RVE) with a novel ranking attention mechanism to mine diverse visual information from CLIP features.
• Incorporated a Ranking Loss (RL) that uses caption quality ranking as a global semantic supervisory signal to enhance the diffusion process and vision-language alignment.

Main Results:

• The RVE effectively mines diverse and discriminative visual information.
• The RL strengthens vision-language semantic alignment by leveraging caption quality ranking.
• The RDT model learns more discriminative visual features precisely aligned with language features through collaborative RVE and RL, and controlled noise diffusion.
• Experimental results show the RDT surpasses existing state-of-the-art image captioning models on benchmark datasets.

Conclusions:

• The proposed Ranking Diffusion Transformer (RDT) effectively addresses limitations in current image captioning models.
• The RDT model demonstrates superior performance in generating descriptive and discriminative captions by enhancing fine-grained visual cue utilization and vision-language alignment.
• The RDT represents a significant advancement in the field of fine-grained image captioning.