MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching.

Area of Science:

• Computer Vision
• Medical Imaging
• Remote Sensing

Background:

• Keypoint detection and description methods excel in single-modality image matching.
• These methods struggle with multimodal data due to descriptor non-robustness against variations.
• Training multimodal methods often requires costly, well-aligned multimodal datasets.

Purpose of the Study:

• To develop a network for learning modality-invariant features for multimodal keypoint description.
• To address the challenge of training multimodal image matching models without paired data.
• To improve the robustness of keypoint descriptors across different imaging modalities.

Main Methods:

• Proposed a modality-invariant feature learning network (MIFNet).
• Introduced novel latent feature aggregation and cumulative hybrid aggregation modules.
• Leveraged pre-trained features from Stable Diffusion models for descriptor enhancement.
• Utilized single-modality training data exclusively.

Main Results:

• MIFNet successfully computes modality-invariant features for multimodal image matching.
• The method demonstrated strong performance across diverse multimodal datasets (retinal, remote sensing).
• Achieved good zero-shot generalization ability without accessing target modality data.

Conclusions:

• MIFNet effectively learns modality-invariant features for multimodal image matching using only single-modality training data.
• The proposed approach overcomes the need for paired multimodal data acquisition.
• MIFNet offers a robust and generalizable solution for multimodal keypoint description.