VSS-SpatioNet: a multi-scale feature fusion network for multimodal image integrations

Area of Science:

• Computer Vision
• Medical Imaging
• Artificial Intelligence

Background:

• Visible-infrared image fusion (vis-ir) is crucial for enhancing diagnostic accuracy in medical imaging and biological analysis.
• Current Convolutional Neural Network (CNN) and Transformer-based methods struggle with computational inefficiencies in modeling global dependencies.

Purpose of the Study:

• To introduce VSS-SpatioNet, a lightweight architecture designed for efficient vis-ir image fusion.
• To address the computational limitations of existing deep learning models in capturing global dependencies.

Main Methods:

• Proposed VSS-SpatioNet architecture replaces Transformer self-attention with a Visual State Space (VSS) module for efficient dependency modeling.
• Employs an asymmetric encoder-decoder structure featuring a multi-scale autoencoder.
• Introduces a novel VSS-Spatial (VS) fusion block for integrated local-global feature representation.

Main Results:

• VSS-SpatioNet achieved state-of-the-art performance on the TNO dataset with high Entropy (En=7.0058) and Mutual Information (MI=14.0116).
• On the RoadScene dataset, the framework surpassed prior methods in gradient-based fusion (SF=0.5712), Piella's metric (Q=0.7926), and average gradient (AG=5.2994).
• The VS strategy demonstrated a significant 18.7% improvement in Mean Gradient on the Harvard Medical dataset compared to FusionGAN, confirming enhanced feature preservation.

Conclusions:

• VSS-SpatioNet proves effective for vis-ir image fusion, offering superior performance and computational efficiency.
• The framework's ability to preserve features is particularly beneficial for precise tissue characterization in medical applications.
• Results validate the potential of VSS-SpatioNet as an advanced tool in medical image analysis.