Re-Attention for Visual Question Answering.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Visual Question Answering (VQA) requires understanding both images and questions.
• Current VQA models focus on associating questions with image objects.
• Answers in VQA contain valuable information for describing visual content.

Purpose of the Study:

• To propose a re-attention framework that utilizes answer information to enhance VQA performance.
• To improve the accuracy of visual attention mechanisms in VQA models.
• To develop a method for correcting visual attention maps using answer-derived insights.

Main Methods:

• A re-attention framework is proposed to leverage answer information for VQA.
• Initial attention weights are learned via word-object similarity in feature space.
• Visual attention maps are reconstructed by re-attending objects based on the answer.
• A gate mechanism controls re-attention contribution using attention map entropy.

Main Results:

• The proposed framework effectively utilizes answer information to refine visual attention.
• Experiments on three benchmark datasets show favorable performance against state-of-the-art methods.
• The re-attention mechanism corrects visual attention maps using answer consistency.

Conclusions:

• The proposed re-attention framework significantly improves VQA performance by incorporating answer information.
• The method demonstrates the potential of using answer-derived context to enhance visual understanding.
• This approach offers a new direction for developing more sophisticated VQA models.