Injecting reasoning into vision-language models via weight-decomposed merging

Model merging offers a promising training-free paradigm for integrating capabilities across pretrained models. While recent advances have demonstrated that merging large language models (LLMs) into vision-language models (VLMs) can endow them with reasoning abilities, we observe that standard weight merging strategies originally designed for structurally homogeneous models often introduce semantic inconsistencies between the vision and language modules. In many cases, this leads to degraded performance after merging, with visual grounding accuracy dropping by more than 50% in some scenarios. In this paper, we propose a weight-decomposed merging framework that explicitly decouples pretrained parameters into directional and magnitude components, and merges them independently to preserve representational alignment. Our method is simple, training-free, and compatible with heterogeneous model architectures. Through extensive empirical evaluation across three VLMs, four reasoning-capable LLMs, and three multimodal reasoning benchmarks, we show that our approach achieves strong and competitive performance in heterogeneous LLM-to-VLM merging settings. The results indicate that weight-decomposed merging provides a stable training-free pathway for reasoning injection while mitigating severe performance degradation observed in several baseline merging methods. The code is available at https://github.com/YC-Zou/Weight-Decomposed-VLM-Merging.