Collaborated With Hallucination: Enhancing Egocentric Grounded Question Answering via Error Demonstrations

Area of Science:

• Computer Science
• Artificial Intelligence
• Robotics

Background:

• Egocentric video question answering (Ego-GQA) requires advanced human-centric reasoning, distinct from third-person perspectives.
• Existing Ego-GQA models struggle with inherent limitations of egocentric context, leading to hallucinations and flawed reasoning.
• Current methods often treat first-person and third-person video understanding identically, neglecting unique egocentric challenges.

Purpose of the Study:

• To propose a novel framework, Collaborated with Hallucination (CoHa), to address hallucinations in Ego-GQA.
• To quantify hallucinations and use them as error demonstrations to constrain model reasoning.
• To encourage Ego-GQA models to ground predictions in visual cues rather than relying on pretraining biases.

Main Methods:

• Utilizing Subjective Logic to quantify uncertainty in unreliable answers.
• Generating diffusion-based noisy visual inputs to amplify hallucinations as error demonstrations.
• Incorporating an interactive refinement module for exploring fine-grained first-person visual cues.

Main Results:

• The CoHa framework effectively quantifies hallucinations and uses them to constrain model reasoning.
• The method steers predictions away from unreliable semantics caused by egocentric thinking limitations.
• Experimental results on benchmarks show CoHa outperforms state-of-the-art Ego-GQA methods.

Conclusions:

• The CoHa framework offers a significant advancement in Ego-GQA by addressing hallucination issues.
• Leveraging hallucinations as error demonstrations improves egocentric reasoning and grounds predictions in visual data.
• The proposed approach enhances the reliability and accuracy of question answering in egocentric videos.