Refining visual token sequence for efficient image captioning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Transformer-based architectures have advanced image captioning (IC) but increased computational complexity and reduced inference speed.
• The visual encoding process is identified as the primary contributor to the computational overhead in IC models.
• Redundancy in visual information, with many irrelevant or low-information regions, presents an opportunity for optimization.

Purpose of the Study:

• To develop an efficient method for reducing the computational overhead in transformer-based image captioning systems.
• To accelerate model inference speed without sacrificing captioning performance.
• To enable flexible trade-offs between accuracy and speed in image captioning.

Main Methods:

• Proposing a knowledge-injection-based visual token Reduction module to estimate token importance and retain a subset.
• Introducing token Fusion and Insertion modules to mitigate semantic loss by reusing discarded tokens and capturing global semantics.
• Deploying a visual token sequence refinement strategy (RFI) within the visual backbone to hierarchically compress token sequences.

Main Results:

• The proposed RFI method effectively reduces the computational overhead of image captioning models.
• Experiments demonstrate that RFI accelerates model inference speed without compromising performance.
• The method allows for adjustable accuracy-speed trade-offs based on specific application requirements.

Conclusions:

• The RFI strategy offers a viable solution for optimizing computationally intensive image captioning models.
• Efficient visual token sequence refinement is key to achieving faster and more practical IC systems.
• The proposed approach balances performance and efficiency, making advanced IC more accessible for real-world applications.