Joint embedding VQA model based on dynamic word vector.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Existing Visual Question Answering (VQA) models often rely on static word vectors, limiting their ability to interpret nuanced language.
• Static word vectors fail to capture context-dependent word meanings and grammatical roles, potentially leading to semantic and grammatical deviations in VQA.

Purpose of the Study:

• To develop a novel joint embedding Visual Question Answering model that utilizes dynamic word vectors.
• To address the limitations of static word vectors in accurately representing word meaning within different linguistic contexts.

Main Methods:

• The proposed model, N-KBSN (none KB-Specific network), employs dynamic word vectors for text characterization.
• Key components include image feature extraction using Faster R-CNN, text feature extraction using ELMo, and feature enhancement via a multi-head attention mechanism.

Main Results:

• The N-KBSN model demonstrated superior performance compared to established models, including those using GloVe (2017 and 2019 winners).
• The integration of dynamic word vectors significantly improved the overall accuracy of the Visual Question Answering task.

Conclusions:

• Dynamic word vectors offer a more effective approach to text characterization in Visual Question Answering models.
• The N-KBSN model represents a significant advancement in VQA, enhancing accuracy through context-aware language understanding.