VMFormer: Visual Clues-Guided Multi-Stage Transformer for Image Captioning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Transformer models excel in image captioning due to self-attention and parallel computation.
• Current methods suffer from a training-inference discrepancy, leading to error accumulation and inaccurate captions.

Purpose of the Study:

• To address the limitations of existing image captioning methods, particularly the training-inference discrepancy and the disruption of Transformer parallelism.
• To introduce VMFormer, a novel Transformer architecture with a Visual-Aware Scheduled Sampling (VASS) module.

Main Methods:

• VMFormer employs a two-stage decoding scheme: an initial self-study stage for candidate token generation and a hybrid stage blending ground-truth and predictions.
• A cognitive-inspired prioritization mechanism focuses on visual keywords (nouns, verbs, attributes) during early training, mimicking human captioning.
• The Visual-Aware Scheduled Sampling (VASS) module is designed as a plug-and-play component, preserving Transformer parallelism.

Main Results:

• VMFormer achieved a CIDEr score of 142.2 on the MSCOCO dataset.
• The model demonstrated consistent improvements when extended to video captioning on MSRVTT and MSVD datasets.
• The VASS module effectively bridges the gap between training and inference conditions, reducing prediction errors.

Conclusions:

• VMFormer, with its VASS module, offers a significant advancement in Transformer-based image and video captioning.
• The proposed method enhances caption quality by mitigating error accumulation and improving visual relevance.
• The plug-and-play nature of VASS allows for easy integration into existing Transformer architectures.