An Ensemble of Generation- and Retrieval-based Image Captioning with Dual Generator Generative Adversarial Network

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Image captioning is crucial for understanding visual content but remains challenging.
• Current methods are either generation-based (synthesizing new captions) or retrieval-based (finding existing captions).
• Both approaches have limitations that hinder optimal performance.

Purpose of the Study:

• To propose a novel image captioning model, EnsCaption, that integrates retrieval-based and generation-based methods.
• To enhance caption quality by combining the strengths of both existing approaches.
• To improve the accuracy and relevance of automatically generated image descriptions.

Main Methods:

• Developed EnsCaption, a model utilizing a dual generator generative adversarial network.
• Incorporated a caption generation model for synthesizing tailored captions.
• Implemented a caption re-ranking model to select the best caption from a pool of generated and retrieved options.
• Employed a discriminator for multi-level difference learning between generated/retrieved and ground-truth captions.

Main Results:

• EnsCaption effectively combines generation-based and retrieval-based image captioning merits.
• The model demonstrated impressive performance on the MSCOCO and Flickr-30K benchmark datasets.
• Experimental results show significant improvements over strong baseline methods.

Conclusions:

• EnsCaption offers a powerful new approach to image captioning by synergizing diverse methods.
• The proposed dual generator adversarial network architecture is effective for enhancing caption generation and ranking.
• The model achieves state-of-the-art or competitive results on standard image captioning benchmarks.