An Efficient Image Fusion Network Exploiting Unifying Language and Mask Guidance

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Image fusion merges data from multiple sensors to enhance image quality and information content.
• Existing methods often rely on complex architectures, downstream tasks, or generative models.
• Guidance from language and semantic masks for image fusion remains underexplored.

Purpose of the Study:

• To investigate the use of language and semantic masks for guiding image fusion.
• To develop a lightweight and efficient image fusion framework.
• To simplify existing complex image fusion methodologies.

Main Methods:

• A bidirectional receptance weighted key value (RWKV) model adapted for image modality using an efficient scanning strategy (ESS).
• A multi-modal fusion module (MFM) for integrating language and mask features.
• A recurrent neural network-like architecture to avoid quadratic-cost attention mechanisms.

Main Results:

• The proposed framework achieved state-of-the-art performance across multiple image fusion tasks.
• Demonstrated effectiveness in visible-infrared, multi-focus, multi-exposure, medical, hyperspectral/multispectral image fusion, and pansharpening.
• The lightweight network design offers computational efficiency.

Conclusions:

• Language and mask guidance offer a promising and simplified approach to image fusion.
• The bidirectional RWKV model with MFM is effective for multi-modal image fusion.
• The framework provides a versatile solution for various challenging image fusion applications.