Visual Cluster Grounding for Image Captioning.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Attention mechanisms are widely used in vision-language tasks like image captioning.
• Existing methods struggle with accurate grounding, leading to object hallucinations and language bias.
• Strict word-object alignment can be suboptimal due to noisy data and appearance variations.

Purpose of the Study:

• To improve both grounding accuracy and captioning performance in vision-language models.
• To develop a novel grounding model that implicitly links words to image evidence.
• To introduce a new metric for evaluating soft attention mechanisms in image captioning.

Main Methods:

• Proposed a novel grounding model with slacked constraints for implicit word-to-image evidence linking.
• The model encourages focus on informative image regions, including discriminative parts or full objects.
• Developed a new quantitative metric to evaluate soft attention by considering all object proposals' contributions.

Main Results:

• The proposed model consistently improves image captioning performance on Flickr30k and MS COCO datasets.
• Demonstrated enhanced grounding accuracy and more grounded image captions.
• The new attention evaluation metric shows better consistency with captioning performance.

Conclusions:

• The novel grounding model effectively improves both grounding and captioning in attention-based architectures.
• Implicit linking with slacked constraints allows for better linguistic and visual relevance.
• The proposed evaluation metric offers a reliable way to assess attention mechanisms.