MIRAGE: Medical image-text pre-training for robustness against noisy environments

Contrastive vision-language pre-training models have achieved significant success on large-scale general multi-modality datasets. However, in the medical domain, the high costs of data collection and expert annotation are likely to result in small-sized and noisy datasets, which can severely limit model performance due to overfitting unreliable data and misrepresenting patterns. To address this challenge, we present MIRAGE, a novel framework designed to handle mismatched false positives and semantically related false negatives during medical image-text pre-training. Cross-entropy-based optimization proves inadequate for noisy contrastive settings, as it tends to fail in distinguishing noisy samples and ends up fitting them, leading to suboptimal representations. To overcome this limitation, we introduce an optimal transport-based contrastive loss that effectively identifies noisy samples leveraging the nearest cross-modality neighbor prior, thereby reducing noisy samples' adverse impact. Additionally, we propose an adaptive gradient balancing strategy that mitigates the influence of gradients from noisy samples. Extensive experiments demonstrate that MIRAGE achieves superior performance across six tasks and 14 datasets, largely outperforming representative state-of-the-art methods. Furthermore, comprehensive analyses on synthetic noisy data are performed, clearly demonstrating the contribution of each component in MIRAGE.