AFSPrompt: An Axiomatic Fuzzy Set Prompt Pipeline for Knowledge-Based VQA

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• In-context learning (ICL) shows promise for knowledge-based visual question answering (VQA), but its performance is highly sensitive to demonstration organization.
• Current VQA research often overlooks the critical role of effective demonstration selection and ranking.

Purpose of the Study:

• To enhance the performance and trustworthiness of knowledge-based VQA by introducing a novel framework for organizing demonstrations.
• To leverage axiomatic fuzzy set (AFS) theory for unsupervised and interpretable demonstration organization in VQA.

Main Methods:

• Proposed AFSPrompt, a train-free framework utilizing AFS theory for example selection and ranking in knowledge-based VQA.
• Filtered irrelevant examples using multimodal embeddings and applied AFS logic to integrate candidate comparison information.
• Employed a compact 7B LLM as a knowledge engine with optimized prompts to reduce reliance on large-scale APIs.

Main Results:

• Demonstrated the effectiveness of AFSPrompt in a lightweight pipeline for knowledge-based VQA tasks.
• Achieved enhanced understanding and trustworthiness in the VQA decision-making process through semantic concept description.
• Validated the approach through extensive evaluations on two benchmark datasets.

Conclusions:

• AFSPrompt offers an effective and lightweight solution for improving knowledge-based VQA by optimizing demonstration organization.
• The integration of AFS theory provides an interpretable and unsupervised method for enhancing AI decision-making in VQA.
• The framework facilitates model deployment by utilizing smaller language models, reducing dependency on large-scale APIs.