RRFNet: a unified network for multimodal image registration and fusion via RepVGG

Infrared and visible image fusion can integrate complementary information from multiple source images to improve the ability of a visual task. Usually, the existing infrared and visible image fusion methods require strictly aligned image pairs as datasets. Meanwhile, the offset of the source image usually affects the quality of the fused image, leading to information loss. In addition, highlighting the target receives more attention, while the visual effect in the fused image is neglected. In this work, we propose a unified network named RRFNet by deploying image registration and image fusion into one end-to-end network. A two-branch registration network registration module is designed to align the input image pairs before the fusion module. Additionally, the autoencoder-based fusion module is designed to integrate image information efficiently. We introduce RepVGG blocks to reduce the number of calculations and parameters after adding the registration module to the network. Especially, we establish the loss function via similarity and average corner error in the registration stage. Meanwhile, the texture loss and structural similarity loss are designed to achieve better visual effects for the fusion results. Numerous qualitative and quantitative experiments are conducted with four state-of-the-art methods on three datasets. The results on five metrics demonstrate that our method realizes effective registration of multimodal images and improves the quality of the fused image simultaneously, which provides a new approach to multimodal image fusion.