Improved Image Caption Rating - Datasets, Game, and Model.

Area of Science:

• Computer Vision
• Natural Language Processing
• Human-Computer Interaction

Background:

• Evaluating image-caption alignment is subjective and lacks standardized metrics.
• Existing datasets for image-caption rating may not fully capture nuanced human judgment.

Purpose of the Study:

• To investigate the subjective nature of image caption quality.
• To develop high-quality datasets for image-caption rating using human feedback.
• To train machine learning models for predicting caption quality.

Main Methods:

• Collected human feedback on image-caption pairs through gamification.
• Validated dataset quality by measuring inter-rater agreement.
• Trained machine learning models on the generated datasets to predict ratings.

Main Results:

• Achieved higher inter-rater agreement compared to previous benchmarks.
• Developed datasets that are more easily learned by machine learning models.
• Demonstrated superior performance of models trained on the new datasets.

Conclusions:

• Gamified human feedback is effective for creating high-quality image-caption rating datasets.
• The developed datasets improve machine learning model performance in predicting caption quality.
• This approach offers a more reliable method for assessing image-caption fit than existing metrics.