Video captioning based on vision transformer and reinforcement learning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Accurate global encoding of visual features is crucial for effective video captioning.
• Existing methods face challenges in comprehensively capturing and describing video content.

Purpose of the Study:

• To propose a novel video captioning method combining Vision Transformer (ViT) and reinforcement learning.
• To enhance the accuracy of video content descriptions by integrating advanced deep learning architectures.

Main Methods:

• Utilized Resnet-152 and ResNeXt-101 for initial video feature extraction.
• Employed the Vision Transformer (ViT) encoding block for sophisticated feature encoding.
• Integrated a Long Short-Term Memory (LSTM) network for generating descriptive captions.
• Fine-tuned the model using reinforcement learning to optimize captioning accuracy.

Main Results:

• Achieved improved performance on the MSR-VTT benchmark dataset.
• Demonstrated significant gains over current mainstream video captioning methods.
• Reported improvements of 2.9% (LEU-4), 1.4% (METEOR), 0.9% (ROUGE-L), and 4.8% (CIDEr-D).

Conclusions:

• The proposed ViT and reinforcement learning-based video captioning method offers superior performance.
• This approach effectively addresses the challenge of global visual feature encoding in video description.
• The findings suggest a promising direction for advancing automated video captioning technologies.