From Image to Pixels: Towards Fine-Grained Medical Vision-Language Models

Area of Science:

• Biomedical Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Current multimodal large language models (MLLMs) in biomedicine lack fine-grained reasoning capabilities for clinical applications.
• Existing models are limited to basic image understanding and textual queries, hindering clinical utility.

Purpose of the Study:

• To develop a comprehensive solution for pixel-level multimodal intelligence in biomedicine.
• To introduce MedPLIB, an end-to-end biomedical MLLM with enhanced visual understanding.
• To create a new benchmark, MeCoVQA, for evaluating spatially-grounded reasoning and diagnostic comprehension.

Main Methods:

• Construction of MeCoVQA, a visual-language benchmark across eight medical imaging modalities and four tasks.
• Introduction of MedPLIB, a biomedical MLLM featuring pixel-level visual understanding and unified modeling for diverse tasks (VQA, querying, grounding, segmentation).
• Design of a task-specialized Mixture-of-Experts (MoE) architecture for MedPLIB, optimized via unified fine-tuning, incorporating retrieval-augmented generation (RAG) and in-context learning (ICL).

Main Results:

• MedPLIB achieves state-of-the-art performance on biomedical vision-language tasks.
• Outperforms existing models by 19.7 and 15.6 mDice in zero-shot pixel-level grounding.
• Demonstrates strong generalization on out-of-distribution medical image segmentation.

Conclusions:

• MedPLIB represents a significant advancement in pixel-level multimodal intelligence for biomedicine.
• The model's performance highlights its clinical utility and strong generalization capabilities.
• Publicly available code and data facilitate further research and development in biomedical AI.