PatchCLIP enables region specific contrastive health record and image joint training with patch embedding loss

Area of Science:

• Computer Vision
• Medical Imaging Analysis
• Artificial Intelligence

Background:

• Vision-Language (VL) models like CLIP excel at zero-shot classification using multimodal self-supervised learning (SSL).
• However, current VL models often lack fine-grained spatial understanding, hindering performance in tasks like medical abnormality localization.
• This limitation necessitates advancements in VL frameworks for precise localization of abnormalities.

Purpose of the Study:

• To introduce Patch-CLIP, a novel VL framework designed to improve spatial understanding.
• To enable effective learning of localization cues by aligning image patch-level embeddings with text embeddings.
• To enhance the performance of abnormality detection in medical imaging.

Main Methods:

• Developed Patch-CLIP, a VL framework incorporating a contrastive loss for aligning image patch embeddings with text embeddings.
• Utilized local patch-level features to encode spatial context, overcoming limitations of global image representations.
• Applied the framework to two Chest X-ray (CXR) datasets for evaluating abnormality detection tasks.

Main Results:

• Achieved state-of-the-art (SOTA) performance across eight abnormality detection tasks on CXR datasets.
• Demonstrated substantial reduction in false positives compared to standard saliency-based methods at similar sensitivity levels.
• Generated precise and interpretable patch prediction maps for key findings.

Conclusions:

• Patch-CLIP significantly enhances fine-grained spatial understanding in VL models.
• The framework provides more accurate and interpretable localization of abnormalities in medical images.
• Patch-CLIP represents a significant advancement for medical imaging analysis and abnormality detection.