Context-aware causal reasoning for explanatory visual question answering

Explanatory Visual Question Answering (EVQA) is a multimodal reasoning task that answers visually relevant natural language questions and generates user-friendly multimodal explanations. Although current EVQA methods can generate grammatically sound and vocabulary appropriate explanations, they still face the challenge of explanatory illusion (i.e., wrong explanations arrive at correct answers). To address this issue, we propose a novel Context-Aware Causal Reasoning (CACR) algorithm. Specifically, we first leverage a Multimodal Large Language Model and design prompting strategies to generate high-quality image descriptions as complementary evidence for helping generate correct explanations. Then, we propose a Structural Causal Model (SCM) to establish the causal relationship among the description, the explanation, and the answer. Finally, we transform the SCM into a deep variational inference network framework that enables causal reasoning to ensure that the explanations are grounded in the visual evidence related to the answer. Extensive experiments show that CACR significantly reduces the risk of explanatory illusion compared to six state-of-the-art methods, outperforming the best baseline by 2.22% on average in terms of explanation quality.