Simignore: Exploring and enhancing multimodal large model complex reasoning via similarity computation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal large language models (MLLMs), including Large Vision-Language Models (LVLMs), often process images by segmenting them into numerous tokens.
• This sequential tokenization approach presents challenges in model interpretability, especially for complex reasoning tasks.

Purpose of the Study:

• To enhance the interpretability and performance of MLLMs in complex reasoning tasks.
• To investigate the information flow dynamics between visual and textual modalities within MLLMs.

Main Methods:

• Utilized Grad-CAM for analyzing image-text interaction dynamics in MLLMs.
• Developed Simignore, an image token reduction technique based on semantic similarity between image and text embeddings.
• Conducted experiments across various MLLM architectures to validate the approach.

Main Results:

• Identified an information flow pattern where interactions converge in early layers and disperse in deeper layers.
• Demonstrated that Simignore effectively ignores semantically irrelevant image tokens.
• Achieved consistent performance improvements in complex reasoning tasks across different MLLM architectures.

Conclusions:

• The Simignore technique enhances MLLM interpretability and reasoning capabilities by focusing on relevant visual-textual information.
• The findings provide a foundation for developing more interpretable and efficient MLLMs.
• The study offers a novel framework for future research in multimodal AI.