LRSP-Fusion: A general image fusion network based on low-rank and sparse priors

Multi-source images captured from the same scene typically contain both common and unique features, which has been widely exploited in specific image fusion tasks. Existing methods typically consider rough separation of common and unique features for a specific fusion task. However, due to the different modal characteristics of modal images in different image fusion tasks, existing feature separation methods are not suitable to general image fusion tasks. To address this issue, we propose an interpretable general image fusion network based on low rank and sparse priors (LRSP-Fusion) for multi-source image fusion tasks. Firstly, based on the principle that image matrices are usually low rank or approximately low rank, a common feature encoder (CFE) is designed based on low-rank matrix factorization theory to extract common features from multi-source images. Then, according to the sparsity of unique features, the optimization models based on sparse constraints are defined, and their corresponding unique feature encoders (UFEs) are designed by unfolding the iterative process of the sparse constraint problem into a neural network to learn the unique features of multi-source images. In addition, to achieve precise fusion of unique and common features, an entropy-based feature fusion block (EFFB) is constructed to achieve the fusion of features from CFE and two UFEs. A large number of experiments on different fusion tasks such as infrared and visible image fusion, medical image fusion, multi-exposure image fusion, and multi-focus image fusion show that our method outperforms some state-of-the-art methods in both objective and subjective aspects.