Image Captioning and Visual Question Answering Based on Attributes and External Knowledge.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Recent advancements in vision-to-language tasks predominantly use Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
• This existing approach bypasses explicit high-level semantic concept representation, directly mapping image features to text.
• A limitation is the inability to answer questions requiring external knowledge beyond the image content.

Purpose of the Study:

• To introduce a novel method for integrating high-level semantic concepts into CNN-RNN models for vision-to-language tasks.
• To demonstrate the efficacy of incorporating external knowledge for advanced visual question answering.
• To achieve state-of-the-art results in image captioning and visual question answering.

Main Methods:

• Proposed a hybrid model combining CNNs and RNNs with explicit high-level concept incorporation.
• Developed a mechanism to integrate external knowledge from a general knowledge base into the visual question answering model.
• Evaluated the model on major benchmark datasets for image captioning and visual question answering.

Main Results:

• Achieved significant improvements over the state-of-the-art in both image captioning and visual question answering.
• Demonstrated the model's capability to answer questions requiring information not present solely within the image.
• The final model attained top reported performance across several benchmark datasets.

Conclusions:

• Integrating high-level concepts and external knowledge substantially enhances vision-to-language models.
• The proposed approach offers a more robust and versatile solution for complex visual question answering.
• This work sets a new benchmark for performance in image captioning and visual question answering.