Structured multi-level knowledge augmentation via small-to-large evidence-guided collaboration for knowledge-based...

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Knowledge-based visual question answering (KB-VQA) necessitates integrating visual information with external knowledge for robust reasoning.
• Existing KB-VQA models struggle with aligning visual content to question intent and resolving ambiguous entity semantics, particularly for complex concepts.

Purpose of the Study:

• To propose an inference-time evidence augmentation framework designed to enhance the performance of frozen large language model (LLM)-based KB-VQA systems.
• To address the limitations of insufficient visual-semantic alignment and entity-level ambiguity in current KB-VQA approaches.

Main Methods:

• Developed a framework utilizing lightweight vision-language models to generate structured textual evidence for a frozen LLM.
• Integrated four modules: question-oriented image information extraction, entity enhancement via sub-questions, candidate-guided answer generation, and contextual exemplar retrieval.
• Focused on systematic construction, refinement, and organization of multimodal evidence prior to LLM inference, without modifying LLM architecture or training.

Main Results:

• Achieved 66.72% accuracy on the OK-VQA dataset and 69.51% on the A-OKVQA dataset, surpassing existing strong baselines.
• Demonstrated robustness, output-format reliability, and acceptable inference costs in supplementary analyses.

Conclusions:

• The proposed evidence augmentation framework effectively improves KB-VQA performance by enhancing multimodal evidence quality for frozen LLMs.
• The method offers a novel strategy for improving LLM-based VQA by focusing on pre-inference evidence construction and refinement.