Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal large language models (MLLMs) excel at visual question answering but struggle with hallucinations.
• Hallucinations in MLLMs are categorized into initial and snowball types, stemming from inadequate token interaction.
• Contextual information is crucial for accurate MLLM inference, yet often overlooked due to outlier token influence.

Purpose of the Study:

• To propose a novel decoding strategy, FarSight, to mitigate hallucinations in MLLMs.
• To enhance in-context inference by improving information propagation between multimodal tokens.
• To reduce attention interference from outlier tokens by optimizing causal masks.

Main Methods:

• FarSight leverages causal masks to establish information propagation between multimodal tokens, inspired by causal inference.
• An attention register structure within the causal mask dynamically allocates attention to prevent diversion to outlier tokens.
• A positional awareness encoding method with a diminishing masking rate improves attention to preceding tokens, particularly for video tasks.

Main Results:

• FarSight significantly reduces hallucinations in multimodal large language models across various benchmarks.
• The plug-and-play strategy demonstrates effectiveness on both image and video-based visual question answering tasks.
• Experiments confirm FarSight's ability to enhance in-context inference by tackling outlier token interference.

Conclusions:

• FarSight offers a versatile and effective solution for mitigating hallucinations in MLLMs.
• Optimizing causal masks and token propagation is key to improving MLLM reliability.
• The proposed method enhances the accuracy and trustworthiness of multimodal AI systems.