Beyond LLaVA-HD: Diving Into High-Resolution Multimodal Large Language Models.

Area of Science:

• Computer Science
• Artificial Intelligence

Background:

• High-resolution vision is crucial for Multimodal Large Language Models (MLLMs) in visual perception and reasoning.
• Current methods for handling high-resolution images in MLLMs are computationally expensive and can dilute global context.

Purpose of the Study:

• To address the computational and contextual limitations of existing high-resolution image processing in MLLMs.
• To propose a novel framework and optimization strategy for efficient and effective high-resolution visual understanding in MLLMs.

Main Methods:

• A mixture of adapters is used to extract global contextual information.
• Learnable query embeddings and a similarity-based selector reduce and select informative local image tokens.
• An alternating training strategy balances global and local feature learning, complemented by a new dataset for local compression training.

Main Results:

• Empirical results show a 'less is more' effect, where fewer, more informative local tokens enhance performance.
• The proposed SliME framework achieves leading performance across benchmarks with limited training data (2 million).
• The alternating training strategy proves more effective than simultaneous end-to-end training.

Conclusions:

• The SliME framework offers an efficient and effective solution for high-resolution image processing in MLLMs.
• Optimized token selection and training strategies are key to improving MLLM performance in visual tasks.
• The proposed method demonstrates significant advancements in MLLM capabilities with reduced computational overhead.