Ask Questions With Double Hints: Visual Question Generation With Answer-Awareness and Region-Reference

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Visual Question Generation (VQG) aims to create human-like questions from images.
• Existing VQG methods struggle with the one-to-many question mapping problem and fail to model complex object relationships or side information interactions.

Purpose of the Study:

• To develop a novel learning paradigm for VQG that generates answer-aware and region-referential questions.
• To address the limitations of existing VQG approaches, specifically the one-to-many mapping issue and inadequate modeling of visual relationships.

Main Methods:

• Proposed a novel learning paradigm using "Double Hints": textual answers and visual regions of interest.
• Developed a self-learning methodology for visual hints without additional human annotations.
• Introduced a double-hints guided Graph-to-Sequence learning framework to model object relationships as a dynamic graph and generate questions.

Main Results:

• The proposed method effectively mitigates the one-to-many mapping issue in VQG.
• The Graph-to-Sequence model successfully captures sophisticated relationships among visual objects.
• Experimental results demonstrate the superiority of the proposed approach over existing methods.

Conclusions:

• The novel learning paradigm and Graph-to-Sequence framework significantly improve visual question generation.
• The "Double Hints" approach enhances question relevance and referentiality.
• The method offers a more effective way to model complex visual scenes for question generation.