On Diversity in Image Captioning: Metrics and Methods.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Image captioning aims to generate descriptive text for images.
• Current evaluation metrics struggle to quantify caption diversity.
• Existing models often prioritize accuracy over diversity, leading to generic captions.

Purpose of the Study:

• To propose novel metrics for evaluating image caption diversity.
• To investigate the trade-off between accuracy and diversity in image captioning models.
• To develop methods for generating more diverse and accurate image captions.

Main Methods:

• Developed a diversity metric based on latent semantic analysis (LSA) kernelized with CIDEr similarity.
• Employed reinforcement learning (RL) strategies, including self-critical learning, to optimize for diversity and accuracy.
• Introduced an ensemble matrix approach and determinantal point processes (DPP) for caption selection and generation.

Main Results:

• The proposed diversity metrics show strong correlation with human evaluations, outperforming mBLEU.
• A significant gap exists between state-of-the-art models and human performance in terms of diversity.
• Balancing cross-entropy loss and CIDEr reward, optimizing diversity metrics, and ensemble methods improved caption diversity and accuracy.

Conclusions:

• Novel diversity metrics effectively evaluate caption quality.
• Reinforcement learning offers viable solutions to bridge the diversity gap in image captioning.
• The proposed methods, particularly ensemble matrix maximization and DPP-inspired selection, significantly enhance both diversity and accuracy.