Multi-task Cross-modal Learning for Chest X-ray Image Retrieval

Area of Science:

• Biomedical informatics
• Artificial intelligence in medicine
• Medical imaging analysis

Background:

• Vision-language foundation models like CLIP and BiomedCLIP provide strong cross-modal embeddings but are not optimized for specific medical retrieval tasks.
• Retrieving clinically relevant radiology reports using chest X-ray (CXR) images requires specialized fine-tuning.

Purpose of the Study:

• To propose and evaluate a multi-task learning framework to fine-tune BiomedCLIP for improved CXR image-text retrieval.
• To enhance the diagnostic sensitivity and clinical relevance of medical image retrieval systems.

Main Methods:

• A multi-task learning framework was developed using BiomedCLIP as the backbone.
• A lightweight MLP projector head was trained with a composite loss function including binary cross-entropy, supervised contrastive loss, and CLIP loss.
• The framework was evaluated on CXR image-text retrieval tasks.

Main Results:

• The fine-tuned model demonstrated more balanced and clinically meaningful performance in both image-to-text and text-to-image retrieval compared to pretrained models.
• t-SNE visualizations showed clearer semantic clustering of normal and abnormal CXR cases, indicating enhanced diagnostic sensitivity.
• The multi-task learning approach improved the model's ability to retrieve relevant medical information.

Conclusions:

• Domain-adaptive, multi-task learning is valuable for advancing cross-modal retrieval in biomedical applications.
• Fine-tuning foundation models like BiomedCLIP with specific medical tasks enhances their clinical utility.
• The proposed framework offers a promising approach for improving medical image retrieval and diagnostic support.