Exploiting Cross-Modal Prediction and Relation Consistency for Semisupervised Image Captioning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Image captioning requires large labeled datasets, posing challenges for manual annotation.
• Real-world scenarios often involve limited labeled data and abundant unlabeled images.
• Existing methods struggle to effectively leverage unlabeled data for training captioning models.

Purpose of the Study:

• To propose a novel semi-supervised image captioning method to address data scarcity.
• To effectively utilize unlabeled images to improve the performance of cross-modal generators.
• To develop a method that constrains generated sentences in the semantic space using raw image input.

Main Methods:

• Cross-modal prediction and relation consistency (CPRC) method is proposed.
• Transforms raw images and generated sentences into a shared semantic space.
• Employs prediction consistency (using raw image predictions as soft labels) and relation consistency (between augmented images and sentences).

Main Results:

• CPRC effectively supervises generated sentences from informativeness and representativeness perspectives.
• The method successfully utilizes unlabeled images in a semi-supervised setting.
• Outperforms state-of-the-art methods on the MS-COCO dataset, achieving at least 6% improvement in CIDEr-D score.

Conclusions:

• CPRC offers a robust solution for semi-supervised image captioning.
• The method enhances caption generation quality by effectively leveraging unlabeled data.
• Demonstrates significant improvements in performance under complex, non-parallel scenarios.