Incorporating modality-specific intensity prior as text prompt for multimodal myocardial pathology segmentation

Accurate myocardial pathology segmentation from multimodal CMR is crucial for myocardial infarction risk assessment and treatment planning. Unlike traditional methods that rely on morphological and geometrical prior for assistance, we propose a novel intensity-guided multimodal image segmentation (I-MMSeg) method for myocardial pathology segmentation in this paper. Specifically, we leverage multimodal large language models (e.g., GPT-4o) to generate modality-specific intensity prior. Such prior capture both the relative intensity orders across different regions and the intrinsic intensity characteristic around the boundary of each pathology, serving as high-level guidance for the segmentation process. To this end, we adopt a CLIP-based model to encode the intensity prior of different modalities, and design two dedicated modules to incorporate them into the segmentation architecture: the intensity-prior-guided cross-modal feature enhancement module and class feature modulation module. The former promotes adaptive recalibration of visual feature and cross-modal feature interaction, while the latter modulates the textual class feature of each modality under the guidance of the intensity prior. Together, they enhance the discriminability of visual feature. Extensive experiments demonstrate that our proposed I-MMSeg method outperforms state-of-the-art myocardial pathology segmentation approaches. The source code and the self-collected MyoPS380 dataset are available at: https://github.com/zzzzzzl24/I_MMSeg.