Using Multiple Instance Learning to Build Multimodal Representations

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Image-text multimodal representation learning is crucial for medical applications like classification and retrieval.
• Existing methods often lack a unified framework for handling diverse data modalities.

Purpose of the Study:

• To establish a connection between multimodal representation learning and multiple instance learning.
• To propose a generic framework for permutation-invariant score functions in multimodal learning.
• To develop a novel contrastive learning approach based on this framework.

Main Methods:

• Connecting multimodal representation learning with multiple instance learning.
• Developing a generic framework for constructing permutation-invariant score functions.
• Deriving a novel contrastive learning method within the proposed framework.

Main Results:

• The proposed framework unifies existing multimodal representation learning approaches.
• The novel contrastive learning method achieves state-of-the-art performance.
• Demonstrated effectiveness across several downstream medical imaging tasks.

Conclusions:

• The established connection provides a new perspective on multimodal representation learning.
• The generic framework offers flexibility and improves performance in medical image analysis.
• The novel contrastive approach represents a significant advancement in the field.