MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities

Area of Science:

• Medical imaging
• Machine learning
• Computer vision

Background:

• Multiple imaging modalities are crucial for medical diagnosis and analysis.
• Limited availability of specific imaging types necessitates advanced data synthesis techniques.
• Existing methods struggle with significant differences between imaging modalities.

Purpose of the Study:

• To develop a novel multi-modal metric learning approach for synthesizing diverse medical images.
• To enable image synthesis even when significant differences exist between imaging modalities.
• To apply the method for synthesizing knee cartilage thickness maps from 2D radiographs.

Main Methods:

• Utilized multi-modal metric learning via image retrieval to create related image embeddings.
• Employed k-nearest neighbor (k-NN) regression for image synthesis based on learned embeddings.
• Validated the approach by synthesizing 3D MR-based cartilage thickness maps from 2D radiographs.

Main Results:

• The proposed retrieval-based synthesis method significantly outperformed direct image synthesis.
• Synthesized cartilage thickness maps retained crucial information for downstream predictive tasks.
• The method demonstrated effectiveness in knee osteoarthritis (KOA) analysis and Kellgren-Lawrence grading (KLG).

Conclusions:

• Multi-modal metric learning and retrieval offer a powerful approach for high-quality medical image synthesis.
• The developed method addresses limitations in modality availability for medical imaging analyses.
• This technique holds promise for improving diagnostic and predictive capabilities in various medical applications.