Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Vision-language models (VLMs) show promise in 2D medical image analysis.
• Extending VLMs to 3D medical imaging is challenging due to computational demands and feature alignment issues.

Purpose of the Study:

• To introduce Med3DVLM, an efficient 3D VLM designed for scalable, multi-task reasoning in clinical applications.
• To address the challenges of analyzing volumetric medical data and aligning 3D spatial features with clinical text.

Main Methods:

• Developed DCFormer, an efficient encoder using decomposed 3D convolutions for capturing fine-grained spatial features.
• Implemented SigLIP, a contrastive learning strategy with pairwise sigmoid loss for improved image-text alignment.
• Utilized a dual-stream MLP-Mixer projector to fuse low- and high-level image features with text embeddings.

Main Results:

• Med3DVLM achieved 61.00% R@1 in image-text retrieval, significantly outperforming M3D-LaMed (19.10%).
• Report generation reached a METEOR score of 36.42% (vs. 14.38%).
• Open-ended VQA scored 36.76% METEOR (vs. 33.58%), and closed-ended VQA achieved 79.95% accuracy (vs. 75.78%).

Conclusions:

• Med3DVLM effectively bridges the gap between 3D medical imaging and clinical language.
• The model demonstrates superior performance across multiple benchmarks, enabling scalable, multi-task reasoning.
• The innovations in Med3DVLM pave the way for advanced clinical applications leveraging multi-modal data.