CTFusion: CNN-transformer-based self-supervised learning for infrared and visible image fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Infrared and visible image fusion (IVIF) integrates information from multi-modal sources.
• Existing fusion methods often require extensive paired image datasets for training.
• Developing efficient IVIF techniques without ground truth data is a significant challenge.

Purpose of the Study:

• To introduce CTFusion, a novel convolutional neural network (CNN)-Transformer framework for IVIF.
• To leverage self-supervised learning for training IVIF models without requiring ground truth fusion images.
• To enhance the extraction of generalized features from infrared and visible images.

Main Methods:

• CTFusion employs an encoder-decoder architecture with a CNN-Transformer-based feature extraction (CTFE) module.
• Self-supervised learning is utilized through a mask reconstruction pretext task tailored for IVIF.
• The framework is designed to model both local and global dependencies in source images.

Main Results:

• CTFusion demonstrated superior performance compared to five traditional and deep learning-based methods.
• Evaluations on three benchmark datasets showed significant improvements in both subjective and objective assessments.
• The self-supervised mask reconstruction task enabled effective learning of image characteristics.

Conclusions:

• CTFusion offers an effective self-supervised approach for infrared and visible image fusion.
• The proposed CNN-Transformer architecture successfully captures essential image features.
• This method advances IVIF by reducing reliance on paired training data and achieving state-of-the-art results.