MultiTaskVIF: Segmentation-Oriented Visible and Infrared Image Fusion via Multi-Task Learning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Visible and infrared image fusion (VIF) is crucial for enhancing image quality.
• Recent VIF methods focus on integrating semantic information for downstream tasks like segmentation.
• Existing segmentation-oriented VIF methods often use complex cascade structures, increasing redundancy.

Purpose of the Study:

• To develop a concise and efficient training framework for segmentation-oriented VIF.
• To integrate semantic information directly into the fusion model during training.
• To reduce network complexity and redundancy in VIF models.

Main Methods:

• Proposed MultiTaskVIF, a multi-task learning (MTL) framework for VIF.
• Introduced a multi-task head decoder (MTH) to inject semantic information into the fusion branch.
• Utilized a hard parameter sharing MTL architecture combined with an attention mechanism.

Main Results:

• The MTH allows existing VIF models to learn semantic features by replacing their decoders.
• The proposed method reduces network complexity and redundancy.
• Enhanced the VIF model's ability to extract segmentation-relevant features, improving fused image quality for segmentation.

Conclusions:

• MultiTaskVIF offers a concise and universal training framework for segmentation-oriented VIF.
• The method effectively integrates semantic information, improving fusion performance for downstream tasks.
• Experimental results validate the effectiveness of the proposed MultiTaskVIF framework.