Uncertainty-Guided Spatiotemporal Consistency Fusion Network for Infrared-Visible Video Fusion under Extremely...

Area of Science:

• Computer Vision
• Image Processing
• Artificial Intelligence

Background:

• Infrared-visible video fusion is crucial for low-light conditions but lacks sufficient high-quality datasets.
• Existing methods struggle with spatiotemporal uncertainty and modality bias in low-light fusion.
• Addressing the scarcity of data and fusion challenges is vital for advancing low-light imaging.

Purpose of the Study:

• To develop a comprehensive dataset for infrared-visible video fusion under extremely low-light conditions.
• To propose an effective fusion network, USCFNet, that addresses spatiotemporal uncertainty and modality bias.
• To improve the quality, detail clarity, and spatiotemporal consistency of fused videos in low-light environments.

Main Methods:

• A new dataset of 4,739 infrared and visible video pairs was created for extremely low-light scenarios.
• An Uncertainty-guided Spatiotemporal Consistency Fusion Network (USCFNet) was developed, incorporating Entropy-Gated SpatioTemporal Attention (EGSTA) modules.
• Difference-Guided Fusion (DGF) and hierarchical mixture-of-experts modules were utilized for adaptive feature fusion and integration.

Main Results:

• USCFNet demonstrated superior performance compared to competing methods on multiple datasets.
• The proposed EGSTA module effectively captures temporal instability and enhances feature spatiotemporal consistency.
• The DGF module improved structural integrity and detail clarity in the fused videos.

Conclusions:

• USCFNet effectively fuses infrared and visible videos under extremely low-light conditions, achieving high spatiotemporal consistency.
• The developed dataset and USCFNet provide valuable resources for research in low-light video fusion.
• The findings highlight the potential of uncertainty-guided attention and adaptive fusion for challenging imaging tasks.