Efficient knowledge distillation and alignment for improved KB-VQA

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge-based visual question answering (KB-VQA) relies on external knowledge for image-related questions.
• Current methods often use Large Language Models (LLMs) but struggle with aligning visual features to knowledge and require significant resources.
• Existing approaches may retrieve irrelevant information, hindering accurate VQA model performance.

Purpose of the Study:

• To propose an efficient approach, EKDA (Efficient Knowledge Distillation and Alignment), for KB-VQA.
• To address limitations of existing LLM-based methods, including poor visual-knowledge alignment and high computational costs.
• To enhance VQA accuracy by effectively integrating visual information with relevant knowledge.

Main Methods:

• Utilizes knowledge distillation with LLaMA as a teacher model for efficient knowledge extraction.
• Employs Graph Neural Networks (GNN) to align visual features with extracted knowledge.
• Focuses on direct alignment between image features and knowledge, rather than just image-text descriptions.

Main Results:

• Achieved state-of-the-art accuracy on the OK-VQA dataset.
• Demonstrated a significant improvement of 6.63% over baseline methods.
• EKDA requires fewer computational resources and avoids complex processes compared to other LLM-based VQA methods.

Conclusions:

• EKDA offers an efficient and effective solution for KB-VQA by improving visual-knowledge alignment.
• The proposed method enhances semantic understanding and model accuracy without high computational overhead.
• This approach represents a significant advancement in leveraging external knowledge for visual question answering.