MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Existing infrared-visible image fusion methods focus on visual quality and downstream task performance.
• Semantic-driven methods incorporate semantic information but lack a macroscopic task-level perspective for reciprocal promotion.
• A gap exists in investigating the interplay between pixel-wise image fusion and cross-modal feature fusion perception.

Purpose of the Study:

• To propose a unified network for simultaneous image fusion and semantic segmentation.
• To explore the reciprocal promotion between image fusion and semantic segmentation tasks.
• To enhance semantic-aware capabilities in image fusion and improve feature-level fusion-based segmentation.

Main Methods:

• Developed a unified network (MAFS) with parallel fusion and segmentation sub-networks.
• Introduced a heterogeneous feature fusion strategy for enhanced semantic awareness.
• Employed a multi-stage Transformer decoder for efficient multi-scale feature aggregation.
• Utilized a dynamic factor for adaptive task weighting in multi-task training.

Main Results:

• Achieved competitive performance compared to state-of-the-art methods in both image fusion and semantic segmentation.
• Demonstrated effective enhancement of semantic-aware capabilities through heterogeneous feature fusion.
• Showcased improved feature-level fusion-based segmentation via knowledge transfer from the fusion sub-network.
• Validated the effectiveness of the dynamic factor for stable multi-task learning.

Conclusions:

• The proposed unified network effectively integrates image fusion and semantic segmentation.
• Reciprocal promotion between tasks leads to improved performance in both domains.
• The MAFS network offers a novel and efficient approach for cross-modal image analysis.