A data-efficient 3D medical vision-language model using only a 2D encoder.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Vision-Language Models (VLMs) show success in 2D medical image analysis.
• Extending VLMs to 3D medical data faces challenges due to limited large-scale annotated datasets and reliance on specialized 3D encoders.

Purpose of the Study:

• To develop a data-efficient framework for 3D medical vision-language tasks.
• To bypass the need for 3D-specific encoders by leveraging pre-trained 2D encoders.

Main Methods:

• A novel pipeline processes volumetric data using a pre-trained 2D vision encoder.
• Key steps include cosine similarity for slice pruning, spatial-frequency fusion for inter-slice correlation, and fine-grained feature injection to preserve details.

Main Results:

• The framework achieved a METEOR score of 50.13 on M3D-Cap report generation.
• It attained 82.90% accuracy on M3D-VQA, outperforming existing models.
• Demonstrated superior performance on public 3D benchmarks.

Conclusions:

• The proposed framework offers a scalable and efficient paradigm for 3D medical vision-language tasks.
• It provides a data-efficient alternative to data-intensive 3D encoders, avoiding the need for 3D-specific pre-training.