Enhancing vision-language model with pretraining for reasoning medical applications

With the rapid advancement of Vision-Language Models (VLMs), there is a growing interest in adapting these models to the medical domain. However, the majority of VLMs treat medical tasks as straightforward question- answering problems, neglecting the need for step-by-step reasoning, which is essential for handling complex medical information and gaining clinical trust. To address these limitations, we develop a Multi-modal Medical Reasoning Model (MMRM), which augments visual-language models (VLMs) by incorporating structured Chain of Thought (CoT) reasoning mechanism for better simulating the real clinical diagnosis process. During the development process, we first propose an Ortho Enhanced Training Framework to optimize the critical visual encoder of the VLM. Secondly, we leverage a Black-box knowledge distillation method to transfer the medical Chain of Thought reasoning capabilities into the large language model, which serves as another key component of the VLM. Finally, we construct a unique multi-modal medical Chain of Thought dataset for training the entire VLM, allowing the model to explicitly learn diagnostic reasoning. Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance on standard medical VQA benchmarks, including SLAKE outperforming existing methods in both diagnostic accuracy and explanation. Critically, the model not only draws conclusions in medical diagnosis but also presents interpretable reasoning pathway strengthens clinical trustworthiness, and ultimately paving the way for deploying AI-assisted healthcare solutions in real-world clinical setting.