Fusion of Visible and Infrared Aerial Images from Uncalibrated Sensors Using Wavelet Decomposition and Deep Learning

Area of Science:

• Computer Vision
• Remote Sensing
• Signal Processing

Background:

• Multi-modal systems leverage specialized sensors optimized for specific wavelengths to gather environmental data.
• Visible-Infrared (VIS-IR) systems are crucial for all-day, all-weather applications but face challenges in direct data correlation due to differing sensor physics.
• Manual image registration is impractical for large datasets, and reliance on calibrated sensors with metadata limits system generalization.

Purpose of the Study:

• To develop an automated end-to-end pipeline for image registration and fusion of VIS-IR data.
• To address the bottleneck of correlating information from VIS-IR sensors without relying on manual registration or camera metadata.
• To propose a novel keypoint-based metric for evaluating the quality of fused images.

Main Methods:

• A recursive crop and scale wavelet spectral decomposition (WSD) algorithm is proposed for automatic extraction of relevant visible data patches.
• Images are registered to a common resolution palette after data extraction.
• A deep neural network (DNN) is employed for the image fusion process.

Main Results:

• The DeepFusion pipeline demonstrates effective image registration and fusion for VIS-IR data.
• Performance is quantified and compared against state-of-the-art classical and DNN methods using open-source and custom datasets.
• A novel keypoint-based metric is introduced for assessing fused output quality, validating the pipeline's efficacy.

Conclusions:

• The proposed DeepFusion pipeline offers an automated and generalized solution for VIS-IR image registration and fusion.
• The WSD algorithm and DNN fusion approach effectively overcome previous limitations in multi-modal data correlation.
• The novel quality metric provides a robust means to evaluate fusion performance in diverse applications.