Multi-View Visual Question Answering with Active Viewpoint Selection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Conventional visual question answering (VQA) relies on single-view images, limiting its applicability in real-world scenarios like human-robot interaction (HRI).
• HRI necessitates handling occluded scenes and varying camera angles, challenges not adequately addressed by single-view VQA.
• Multi-view settings are crucial for VQA in HRI, but efficiently acquiring necessary views remains a challenge.

Purpose of the Study:

• To propose a novel framework for iterative visual question answering (VQA) that actively observes scenes from multiple viewpoints.
• To enhance VQA performance in complex environments by enabling iterative scene observation until sufficient information is gathered.
• To reduce the number of required observations and camera movements for effective VQA in multi-view settings.

Main Methods:

• Developed a framework for iterative scene observation to gather visual information for answering questions.
• Implemented an active observation strategy to select optimal viewpoints for data acquisition.
• Constructed a new multi-view VQA dataset using real-world images.

Main Results:

• The proposed framework achieved performance comparable to state-of-the-art methods in question answering.
• Significantly reduced the number of required observation viewpoints compared to existing approaches.
• Demonstrated the framework's ability to learn optimal viewpoint selection, minimizing camera movements.
• Achieved high accuracy (94.01%) on an unseen real image dataset.

Conclusions:

• The iterative, multi-view VQA framework effectively addresses limitations of single-view methods in complex environments like HRI.
• The framework's active viewpoint selection optimizes data acquisition, leading to improved efficiency and reduced computational cost.
• The developed dataset and framework show strong potential for advancing VQA research in real-world applications.