Injecting reasoning into vision-language models via weight-decomposed merging.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Model merging integrates capabilities from pretrained models without fine-tuning.
• Merging large language models (LLMs) into vision-language models (VLMs) can impart reasoning abilities.
• Standard merging techniques often cause semantic inconsistencies and performance degradation in heterogeneous models.

Purpose of the Study:

• To propose a novel weight-decomposed merging framework for heterogeneous LLM-to-VLM integration.
• To address semantic inconsistencies and performance degradation in current model merging approaches.
• To enable training-free reasoning injection into VLMs.

Main Methods:

• Developed a weight-decomposed merging framework that separates parameters into directional and magnitude components.
• Independently merged these components to maintain representational alignment between vision and language modules.
• The method is training-free and compatible with heterogeneous model architectures.

Main Results:

• Achieved strong and competitive performance in heterogeneous LLM-to-VLM merging across multiple benchmarks.
• Demonstrated significant mitigation of performance degradation compared to baseline merging methods.
• Preserved visual grounding accuracy, avoiding the >50% drop seen in other methods.

Conclusions:

• Weight-decomposed merging offers a stable, training-free pathway for enhancing VLMs with LLM reasoning capabilities.
• The proposed method effectively preserves representational alignment, leading to improved multimodal reasoning.
• This approach provides a practical solution for integrating diverse pretrained models without performance loss.