SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusion

Area of Science:

• Computer Vision
• Image Processing
• Machine Learning

Background:

• Visible and infrared image fusion (VIF) is crucial for tasks like scene segmentation and object detection.
• Traditional VIF methods focus on fused image quality, while application-oriented methods optimize downstream task performance.
• Application-oriented VIF methods typically require labor-intensive, labeled datasets for training.

Purpose of the Study:

• To develop a self-supervised training framework for segmentation-oriented VIF (SSVIF) that eliminates the need for labeled downstream task data.
• To enable VIF models to learn high-level semantic features without segmentation labels.

Main Methods:

• Proposed a self-supervised training framework (SSVIF) for segmentation-oriented VIF.
• Introduced a novel self-supervised task: cross-segmentation consistency, leveraging feature-level and pixel-level fusion segmentation consistency.
• Designed a two-stage training strategy and dynamic weight adjustment for effective joint learning.

Main Results:

• Extensive experiments on public datasets validated the effectiveness of the SSVIF framework.
• SSVIF, trained on unlabeled data, outperformed traditional VIF methods.
• SSVIF achieved performance comparable to supervised segmentation-oriented VIF methods.

Conclusions:

• The proposed SSVIF framework offers an effective self-supervised approach for segmentation-oriented VIF.
• SSVIF significantly reduces the data acquisition burden by eliminating the need for labeled datasets.
• This work advances self-supervised learning in image fusion for improved performance in downstream vision tasks.