Deep learning-driven image captioning: Progress through transformers and large language models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Traditional Convolutional Neural Network-Recurrent Neural Network (CNN-RNN) hybrids and existing transformer models have limitations in image captioning.
• Achieving robust multimodal alignment and enhancing caption diversity remain key challenges in the field.

Purpose of the Study:

• To propose a novel deep learning model for image captioning using an advanced vision transformer architecture and a powerful Large Language Model (LLM).
• To improve the alignment between linguistic context and visual features through a unique cross-attention mechanism.

Main Methods:

• Development of a novel deep learning architecture integrating a vision transformer with an LLM.
• Implementation of a unique cross-attention mechanism for deep alignment between visual and linguistic features.
• Extensive evaluation on benchmark datasets including MSCOCO, Flickr30K, and NoCaps.

Main Results:

• The proposed model demonstrates significant improvements over traditional and existing transformer-based approaches.
• Achieved state-of-the-art performance comparable to leading methods like GIT, BLIP-2, and CoCa on MSCOCO, Flickr30K, and NoCaps.
• Specific metrics on MS COCO include BLEU-4 (0.495), METEOR (0.390), and CIDEr (1.32).

Conclusions:

• The novel architecture sets a new performance benchmark for image captioning systems.
• The fusion strategy proves efficient, enabling more precise, contextually rich, and human-like image descriptions.
• This work advances multimodal AI systems, supporting Sustainable Development Goals 9 and 4.