Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question...

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Medical Visual Question Answering (VQA) requires integrating visual and textual data, but lacks large-scale annotated datasets for end-to-end training.
• Existing approaches pre-train models on image captioning datasets before fine-tuning for VQA tasks.

Purpose of the Study:

• To develop a novel pre-training method, Cross-Modal pre-training with Multiple Objectives (CMMO), to enhance medical VQA model performance.
• To improve the alignment between medical image features and textual concepts for better multi-modal reasoning.

Main Methods:

• Utilized public medical image captioning datasets for self-supervised pre-training of multimodal models.
• Implemented CMMO, incorporating masked image modeling, masked language modeling, image-text matching, and contrastive learning.
• Fine-tuned the pre-trained models on downstream medical VQA tasks.

Main Results:

• The CMMO method achieved state-of-the-art results on three public medical VQA datasets: VQA-RAD, PathVQA, and SLAKE.
• Demonstrated absolute performance improvements of 2.6% on VQA-RAD, 0.9% on PathVQA, and 4.0% on SLAKE.
• Ablation studies and attention map visualizations confirmed the method's effectiveness and interpretability.

Conclusions:

• The proposed CMMO method effectively addresses the challenge of limited annotated medical VQA data.
• CMMO enhances multi-modal interactions and feature representation alignment, leading to superior medical VQA performance.
• The release of code and pre-trained weights facilitates further research in medical VQA.