Progressive Modality Cooperation for Multi-Modality Domain Adaptation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Domain adaptation is crucial for applying models to new data distributions.
• Multi-modality data (e.g., RGB, depth) offers richer information but poses adaptation challenges.
• Existing methods struggle with missing modalities in target domains.

Purpose of the Study:

• To develop a generic framework for multi-modality domain adaptation (MMDA).
• To address challenges in MMDA, including missing modalities (MMDA-PI).
• To enhance cross-domain visual recognition by leveraging multiple data sources.

Main Methods:

• Proposed Progressive Modality Cooperation (PMC) framework for knowledge transfer.
• Introduced modules for selecting reliable pseudo-labeled target samples in MMDA.
• Developed PMC with Privileged Information (PMC-PI) using a Multi-Modality Data Generation (MMG) network for MMDA-PI.
• MMG employs adversarial learning and semantic conditioning to generate missing modalities.

Main Results:

• Demonstrated effectiveness of PMC across three image and eight video datasets.
• Achieved significant improvements in multi-modality cross-domain visual recognition tasks.
• Validated performance under both MMDA and MMDA-PI settings.

Conclusions:

• PMC framework offers a robust solution for multi-modality domain adaptation.
• The proposed MMG network effectively handles missing modalities.
• The approach significantly advances cross-domain visual recognition capabilities.