Attention re-alignment in multimodal large language models via intermediate-layer guidance

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal large language models (MLLMs) excel at visual question answering (VQA).
• MLLMs often fail to focus on fine-grained visual details during image analysis.
• This limitation stems from language priors diluting visual attention in deeper model layers.

Purpose of the Study:

• To address the issue of MLLMs neglecting fine-grained visual details.
• To enhance the visual grounding capabilities of existing MLLMs.
• To improve the accuracy and sensitivity of MLLMs in VQA tasks.

Main Methods:

• Proposing a plug-and-play Attention Re-alignment (ARA) module.
• Conducting layer-wise analysis of attention distributions in image-centric heads.
• Employing a confidence-aware layer selection based on attention peak and entropy.
• Dynamically aggregating attention maps from informative layers to guide semantic mask generation.

Main Results:

• The ARA module effectively enhances suppressed visual grounding.
• Semantic masks generated using ARA emphasize salient visual regions and suppress noise.
• Consistent performance improvements observed across multiple VQA benchmarks.
• Demonstrated effectiveness in improving MLLMs' sensitivity to visual details.

Conclusions:

• The ARA module is a viable solution for improving MLLMs' visual detail perception.
• ARA can be seamlessly integrated into existing MLLM architectures.
• Enhanced visual grounding via ARA leads to superior performance in VQA tasks.