Novel concept-based image captioning models using LSTM and multi-encoder transformer architecture.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Image captioning models traditionally rely on text-based topic modeling.
• Text-only topic modeling overlooks crucial image semantic information.
• Existing methods struggle to fully capture image context for accurate descriptions.

Purpose of the Study:

• To propose novel image captioning models leveraging concept modeling.
• To integrate image semantic information directly into the captioning process.
• To enhance the accuracy and contextual relevance of generated image descriptions.

Main Methods:

• Developed two concept-based image captioning models.
• Model 1: Utilizes Long Short-Term Memory (LSTM) as a decoder.
• Model 2: Employs a novel multi-encoder transformer architecture.
• Incorporated concept modeling to extract information directly from images.

Main Results:

• Proposed models demonstrated superior performance compared to existing methods.
• Achieved improved image descriptions by incorporating visual concepts.
• Demonstrated reduced computational complexity in the proposed models.
• Evaluated using standard metrics on Microsoft COCO and Flickr30K datasets.

Conclusions:

• Concept modeling significantly enhances image captioning by integrating visual semantics.
• The proposed LSTM and transformer-based models offer effective solutions for accurate image description.
• The novel approach provides a more comprehensive understanding of image content for caption generation.