Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays

Area of Science:

• Medical Informatics
• Artificial Intelligence
• Computer Vision

Background:

• Multimodal learning from paired medical images and clinical text is crucial for medical data informatics.
• Effective cross-modal alignment is essential for scalable analysis and retrieval of medical data.
• Chest radiography vision-language pretraining faces challenges due to heterogeneous radiology reports (abbreviations, varied styles).

Purpose of the Study:

• To develop a domain-adapted bidirectional large language model text encoder for chest radiograph reports.
• To improve image-text alignment in a dual-tower contrastive vision-language framework.
• To create robust and generalizable text embeddings for chest radiography reports.

Main Methods:

• Trained a domain-adapted bidirectional large language model text encoder using masked token prediction and supervised contrastive learning.
• Integrated the encoder into a dual-tower contrastive vision-language framework with parameter-efficient adaptation.
• Utilized 1.6 million paired studies from public datasets and a hospital cohort.

Main Results:

• Improved bidirectional retrieval accuracy and external generalization across large datasets.
• Achieved GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I.
• Reduced performance degradation when incorporating abbreviation-rich, impression-only hospital reports into training.

Conclusions:

• Robust cross-modal embeddings enable scalable retrieval and multimodal representation learning from routine clinical data.
• The proposed domain-adapted encoder enhances multimodal learning from heterogeneous medical reports.
• This approach facilitates biomedical and health informatics applications using clinical data.