Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction.

Area of Science:

• Machine Learning
• Computer Vision
• Deep Learning

Background:

• Multimodal fusion and multitask learning are key areas in machine learning, yet existing methods struggle to balance common and specific information integration.
• Current approaches often treat multimodal fusion and multitask learning separately, despite their inherent relationship.

Purpose of the Study:

• To introduce a novel, self-adaptive, and parameter-free network, Channel-Exchanging-Network (CEN), for effective multimodal fusion and multitask learning in dense image prediction.
• To address the challenge of integrating cross-modal/task information while preserving modality/task-specific patterns.

Main Methods:

• Proposes Channel-Exchanging-Network (CEN), which adaptively exchanges channels between subnetworks of different modalities.
• Utilizes channel importance, measured by Batch-Normalization (BN) scaling factors, to guide the self-guided channel exchange process.
• Applies CEN to multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning scenarios.

Main Results:

• Demonstrates the effectiveness of CEN in semantic segmentation using RGB-D data and image translation across multiple domains.
• Achieves superior performance compared to state-of-the-art methods in various dense image prediction tasks.
• Ablation studies confirm the advantage and contribution of each proposed component within the CEN framework.

Conclusions:

• CEN provides a unified and effective framework for both multimodal fusion and multitask learning in dense image prediction.
• The proposed channel exchange mechanism offers a robust solution for balancing shared and specific feature learning.
• The method's parameter-free and self-adaptive nature makes it broadly applicable and efficient.