Wisdom Is Knowledge Combined with Intellect: Knowledge-Embedded Hypergraph-of-Thought Reasoning for Visual Abductive...

For a still image, humans can abductively ratiocinate various probable inferences and hypotheses beyond the given image from visual clues scattered throughout a pictorial scene. However, achieving such a task, i.e., visually abductive ratiocination, is challenging for models because of the limitations of observations, knowledge, and their synthesis process. Therefore, we propose a novel multimodal model: knowledge-embedded hypergraph-of-thought reasoning. First, it represents a physical scene as a semantic graph and a hypergraph to capture the underlying intra- and inter- visual -linguistic relationships, thereby overcoming the limitations of observations derived from the spatial domain. Then, a multimodal knowledge hypergraph is constructed to discriminate and associate scene-related multimodal multigranular concepts with a hierarchical structure, leveraging web resources. This approach overcomes the limitations of lacking world knowledge. Finally, a multimodal hypergraph-of-thought reasoning paradigm is devised based on the semantic and knowledge hypergraphs to model the higher-order relationships and interaction mechanisms between the graphs for conducting multihop reasoning, thus overcoming the limitations of semantics and knowledge being separate and independent. Our method reflects the philosophy of "wisdom is knowledge combined with intellect" in the sense that reasoning models including multimodal knowledge and semantic representations are more robust than those that do not synthesize them. The experimental results show that our model achieves significantly greater performance than state-of-the-art methods do.