UNK-VQA: A Dataset and a Probe Into the Abstention Ability of Multi-Modal Large Models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Building trustworthy AI systems requires Visual Question Answering (VQA) models to abstain from answering unanswerable questions.
• Existing VQA research has largely overlooked the critical aspect of model abstention for unanswerable queries.

Purpose of the Study:

• To address the research gap in VQA model abstention by introducing the UNK-VQA dataset.
• To create a benchmark dataset specifically designed to challenge VQA models with questions they cannot answer.
• To evaluate and improve the trustworthiness of AI systems through enhanced VQA abstention capabilities.

Main Methods:

• Augmenting existing VQA data through deliberate image or question perturbations, maintaining close semantic similarity.
• Ensuring perturbations make unanswerable question identification challenging, differentiating UNK-VQA from datasets with simple image replacements.
• Evaluating zero- and few-shot performance of multi-modal large models on the UNK-VQA dataset and proposing a novel method to handle unanswerable questions.

Main Results:

• Emerging multi-modal large models exhibit significant limitations when evaluated on the UNK-VQA dataset.
• The proposed straightforward method offers a potential solution for tackling unanswerable questions in VQA.
• The UNK-VQA dataset effectively highlights the challenges in VQA model abstention.

Conclusions:

• The UNK-VQA dataset serves as a valuable benchmark for advancing VQA model abstention capabilities.
• Improving VQA model abstention is essential for developing more trustworthy AI systems.
• The availability of the UNK-VQA dataset will foster further research in VQA model trustworthiness and abstention.