Boosting Faithful Multi-Modal LLMs via Complementary Visual Grounding.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal Large Language Models (MLLMs) excel at vision-language tasks but suffer from hallucinations, generating text inconsistent with visual input.
• Current methods for mitigating hallucinations are often post-hoc, data-intensive, or computationally expensive.
• Key factors contributing to MLLM hallucination include insufficient visual context and progressive textual drift during generation.

Purpose of the Study:

• To address the core issues of insufficient visual context and progressive textual drift in MLLMs.
• To introduce a novel framework, Complementary Visual Grounding (CVG), that enhances factual consistency in MLLM outputs.
• To develop a method that leverages the intrinsic architecture of MLLMs without external dependencies.

Main Methods:

• The proposed Complementary Visual Grounding (CVG) framework disentangles visual context into query-relevant complementary branches.
• CVG ensures sustained visual grounding throughout the auto-regressive text generation process.
• It contrasts the output distributions from these branches to generate factually grounded responses.

Main Results:

• CVG achieves state-of-the-art performance on hallucination benchmarks.
• The framework demonstrates effectiveness across various MLLM architectures and scales.
• Experiments confirm CVG's ability to produce faithful responses by maintaining visual grounding.

Conclusions:

• Complementary Visual Grounding (CVG) effectively mitigates hallucination in Multimodal Large Language Models (MLLMs).
• The CVG framework offers an intrinsic, efficient solution by improving visual context utilization and preventing textual drift.
• This approach advances the reliability and factual accuracy of MLLMs in vision-language tasks.