Variational Transformer: A Framework Beyond the Tradeoff Between Accuracy and Diversity for Image Captioning.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Caption generation models face a trade-off between accuracy and diversity.
• Human annotations, often used for training, have suboptimal accuracy for machine learning.
• Existing methods struggle to enhance diversity without compromising accuracy.

Purpose of the Study:

• To propose a novel variational transformer (VaT) framework for enhanced image captioning.
• To address the conflicting relationship between accuracy and diversity in generated captions.
• To achieve simultaneous improvements in both accuracy and diversity.

Main Methods:

• Developed a variational transformer (VaT) framework integrating "invisible information prior (IIP)" and "auto-selectable Gaussian mixture model (AGMM)" for high accuracy.
• Incorporated the "range-median reward (RMR)" baseline within a reinforcement learning process to boost diversity.
• Enabled the encoder to learn precise linguistic information and object relationships for scene understanding.

Main Results:

• Achieved simultaneous improvements in accuracy (up to 1.1%) and diversity (up to 4.8%) over state-of-the-art methods.
• Demonstrated performance closest to human annotations in semantic retrieval (50.3 vs. 50.6).
• The proposed VaT framework effectively balances accuracy and diversity.

Conclusions:

• The VaT framework offers a viable solution to the accuracy-diversity dilemma in image captioning.
• The method shows potential for industrial application due to its strong performance.
• This research advances the field of natural language generation for image descriptions.