Fully-Connected Transformer for Multi-Source Image Fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multi-source image fusion aims to enhance imaging quality by combining information from multiple sources.
• Existing self-attention transformer methods capture spatial and channel similarities but face challenges in fully integrating diverse information.
• Developing advanced mechanisms is crucial for effectively exploiting correlations within multi-source images.

Purpose of the Study:

• To propose a novel generalized self-attention mechanism for improved multi-source image fusion.
• To introduce a Fully-Connected Self-Attention (FCSA) method leveraging multilinear algebra.
• To develop a unified network model, the FC-Former, capable of handling various fusion tasks.

Main Methods:

• Developed a generalized self-attention mechanism based on multilinear algebra.
• Introduced a novel Fully-Connected Self-Attention (FCSA) method to exploit local and non-local correlations.
• Proposed a multi-source image representation integrated into the FCSA framework within an optimization problem, forming the FC-Former network.

Main Results:

• The proposed FC-Former network effectively integrates information from multi-source images.
• Demonstrated robust and superior performance compared to state-of-the-art fusion methods.
• Showcased the capability of faithfully preserving information during the fusion process.

Conclusions:

• The FC-Former, utilizing a generalized self-attention mechanism, offers a unified approach to multi-source image fusion.
• The novel FCSA method enhances the exploitation of domain-specific correlations.
• The proposed method achieves superior performance and information preservation in image fusion tasks.