Multi-Modality Image Fusion and Object Detection Based on Semantic Information.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Existing deep learning methods for infrared and visible image fusion (IVIF) often neglect transmission characteristics, leading to information degradation.
• Current fusion techniques may retain redundant or invalid information, hindering downstream tasks like target detection.

Purpose of the Study:

• To develop an end-to-end deep learning network for accurate infrared and visible image fusion.
• To enhance the extraction of effective information from both image modalities without omission or redundancy.
• To improve the performance of subsequent target detection tasks using fused images.

Main Methods:

• Proposed a multi-level structure search attention fusion network guided by semantic information.
• Introduced neural architecture search (NAS) to optimize network design.
• Developed a novel multilevel adaptive attention module (MAAB) to retain essential features and remove irrelevant information.

Main Results:

• The proposed fusion network achieved advanced performance in subjective and objective evaluations on the M3FD dataset.
• Demonstrated improved mean average precision (mAP) by 0.5% in object detection tasks compared to FusionGAN.
• Effectively retained characteristic information from both infrared and visible images while eliminating useless data.

Conclusions:

• The developed fusion network provides a robust solution for infrared and visible image fusion.
• The integration of NAS and MAAB significantly enhances fusion quality and downstream task performance.
• The proposed method offers a reliable approach for generating fused images that are highly beneficial for target detection.