Defying Distractions in Multimodal Tasks: A Novel Benchmark for Large Vision-Language Models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Large Vision-Language Models (LVLMs) are susceptible to "multimodal distractibility."
• Irrelevant visual or textual inputs degrade LVLM reasoning consistency and reliability.
• Existing evaluation methods do not adequately capture these vulnerabilities.

Purpose of the Study:

• To introduce a comprehensive framework for diagnosing, evaluating, and mitigating multimodal distractibility in LVLMs.
• To develop novel methods for measuring reasoning stability beyond standard accuracy.
• To propose a mechanism for proactively filtering distractions during inference.

Main Methods:

• Development of the large-scale IR-VQA benchmark to identify LVLM vulnerabilities.
• Introduction of Positive Consistency (PC) and Negative Consistency (NC) metrics for assessing reasoning stability.
• Implementation of the Relevance-Gated Multimodal Routing (RGMR) mechanism for dynamic distraction filtering.

Main Results:

• State-of-the-art LVLMs show significant consistency drops on the IR-VQA benchmark.
• Fine-tuning on IR-VQA and employing RGMR substantially enhance model robustness.
• RGMR proves effective in improving reliability where standard prompting methods fail.

Conclusions:

• Multimodal distractibility is a critical challenge for current LVLMs.
• The proposed framework, metrics, and RGMR mechanism offer a viable path towards more reliable multimodal systems.
• Further analysis of model behavior under distraction provides insights for future research.