On Distinctive Image Captioning via Comparing and Reweighting.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Current image captioning models excel in standard metrics (BLEU, CIDEr, SPICE) but often produce generic captions lacking distinctiveness.
• Over-reliance on overlap-based metrics leads to common word usage and similar captions for visually similar images.
• Existing training methods treat all human annotations equally, potentially hindering the generation of unique and informative captions.

Purpose of the Study:

• To improve the distinctiveness of automatically generated image captions.
• To address the limitation of generic captions by incorporating information from similar images.
• To develop a new metric for evaluating caption distinctiveness.

Main Methods:

• Proposed a novel distinctiveness metric, CIDEr between-set (CIDErBtw), to evaluate caption uniqueness against similar images.
• Introduced a reweighting strategy for ground-truth captions during training, emphasizing distinctiveness.
• Integrated a long-tailed weight strategy to highlight informative rare words and used similar image captions as negative examples.

Main Results:

• The proposed CIDErBtw metric revealed non-uniform distinctiveness among human annotations in the MSCOCO dataset.
• Reweighting captions based on distinctiveness and incorporating rare words significantly improved caption uniqueness.
• The approach enhanced both distinctiveness (CIDErBtw, retrieval metrics) and accuracy (CIDEr) across various baseline models.

Conclusions:

• The proposed method effectively enhances image caption distinctiveness beyond traditional overlap-based metrics.
• Reweighting training data and focusing on rare words are crucial for generating more unique and informative captions.
• Results were validated through extensive experiments and a user study, confirming improvements in both accuracy and distinctiveness.