IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-Training

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Medical Vision-Language Pre-training (VLP) typically extracts features from clinical reports and images.
• Existing VLP methods often overlook the inherent hierarchical structure of clinical reports (e.g., 'findings' and 'impressions').
• Current approaches oversimplify reports into single entities or fragmented tokens, losing valuable structural information.

Purpose of the Study:

• To propose a novel clinical prior guided VLP framework, IMITATE.
• To leverage the hierarchical structure of medical reports for improved vision-language alignment.
• To enhance VLP by incorporating clinical prior knowledge through a specialized contrastive loss.

Main Methods:

• Developed IMITATE, a VLP framework with hierarchical vision-language alignment.
• Derived multi-level visual features from chest X-ray (CXR) images.
• Separately aligned visual features with descriptive ('findings') and conclusive ('impressions') text segments.
• Introduced a clinical-informed contrastive loss for cross-modal learning.

Main Results:

• IMITATE significantly outperformed baseline VLP methods.
• The model demonstrated superior performance across six diverse datasets.
• Effectiveness was validated across five different medical imaging downstream tasks.
• Experimental results confirmed the benefits of utilizing hierarchical structures in medical reports for VLP.

Conclusions:

• The proposed IMITATE framework effectively utilizes the hierarchical structure of clinical reports for medical VLP.
• Hierarchical alignment and clinical-informed contrastive learning enhance VLP model performance.
• The study highlights the importance of incorporating domain-specific structural information in medical VLP.