Focus on the essentials: Learning to attend to the most critical information for visual question answering

Most visual question answering models suffer from severe language bias issues, which capture the biases between the question and answer rather than genuine multimodal reasoning. To tackle this issue, some researchers have proposed the counterfactual synthesis strategy, which aims to guide models to focus on critical regions and words, thereby reducing language bias. However, these models tend to obscure irrelevant information, failing to attend to the most critical visual regions or words, which can inadvertently introduce new biases. Moreover, they typically overlook the inherent difficulty disparities across different question types, preventing simultaneous optimization for all question types and ultimately impairing overall generalization performance. To this end, we propose a novel model, called FOE, designed to help the model focus on the essentials. We first employ innovative counterfactual generators to create minimal perturbations - masking critical image regions or pivotal question words - to expose spurious correlations. Subsequently, we propose a dynamic answer penalty mechanism that adaptively suppresses biases based on question types. Finally, we apply contrastive learning in the cross-modal joint embedding space to differentiate between factual and counterfactual samples more effectively. This enables the model to better capture subtle distinctions between them, leading to improved attention to critical visual regions and linguistic features. Extensive experimental results substantiate the effectiveness of FOE, especially when integrated with the LMH module, yielding superior performance across the VQA v2, VQA-CP v2, GQA-OOD and VQA-CE datasets.