Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Visual Question Answering (VQA) requires integrating visual and textual information.
• Existing VQA models face challenges in fine-grained feature representation, multimodal fusion, and answer prediction.
• Addressing these challenges is crucial for advancing VQA capabilities.

Purpose of the Study:

• To develop a unified deep neural network (DNN) architecture for improved Visual Question Answering (VQA).
• To enhance fine-grained feature representations for images and questions.
• To achieve effective multimodal feature fusion and accurate answer prediction.

Main Methods:

• A coattention mechanism within a DNN architecture for joint image and question attention.
• A generalized multimodal factorized high-order pooling (MFH) approach for effective multimodal feature fusion.
• Kullback-Leibler divergence as a loss function for precise answer prediction, considering answer correlations.

Main Results:

• The coattention mechanism effectively reduces irrelevant features and enhances discriminative representations.
• The MFH approach significantly improves multimodal feature fusion, outperforming state-of-the-art methods.
• The proposed model achieved state-of-the-art results on large-scale VQA datasets and secured runner-up in VQA Challenge 2017.

Conclusions:

• The integrated DNN architecture with coattention, MFH, and KL divergence offers a superior solution for VQA.
• The model demonstrates strong performance in understanding complex visual and textual interactions.
• This work advances the field of VQA, providing a robust framework for future research and applications.