Bilinear Graph Networks for Visual Question Answering.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Classical bilinear attention networks (BANs) for visual question answering (VQA) extract joint representations but struggle with complex relational reasoning between question words and image objects.
• Existing methods often fail to fully explore the intricate dependencies between linguistic elements and visual entities, limiting performance on sophisticated VQA tasks.

Purpose of the Study:

• To develop an enhanced graph-based approach for VQA that effectively models inter-object and word-object relationships.
• To improve the capacity for multistep reasoning in VQA systems by leveraging graph structures.

Main Methods:

• Introduced bilinear graph networks (BGNs) to model contextual relationships within joint embeddings of words and objects.
• Investigated two graph types: image-graph for transferring object features to query words and question-graph for inter-object information exchange.
• Developed a cooperative framework where image-graph and question-graph work together to capture object dependencies.

Main Results:

• The proposed bilinear graph networks demonstrated improved handling of complex questions on the VQA v2.0 validation dataset.
• The best single model achieved state-of-the-art accuracy of 72.56% on the VQA test-std set.
• The method was recognized as a top-two entry in the VQA Challenge 2020.

Conclusions:

• Bilinear graph networks offer a powerful framework for visual question answering by explicitly modeling relationships and dependencies.
• This graph-based approach significantly enhances multistep reasoning capabilities, leading to superior performance on complex VQA tasks.
• The method represents a significant advancement in VQA, achieving state-of-the-art results and competitive performance in a major challenge.