GText-IF: Leveraging Text-Driven Semantics for Degradation-Aware Image Fusion

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Current image fusion methods struggle with degraded source images, leading to artifacts in fused outputs.
• Existing techniques lack adaptive mechanisms to mitigate degradations during the fusion process.
• This limits the comprehensive information representation in the final fused image.

Purpose of the Study:

• To propose an innovative degradation-aware image fusion framework (GText-IF) that leverages text-driven semantics.
• To extend image fusion to text and image fusion, exploring deep text-image coupling for guided fusion.
• To enhance the quality and information representation of fused images, especially in the presence of source image degradations.

Main Methods:

• Incorporated textual information as semantic guidance using vision-language models and human prior knowledge.
• Utilized semantic segmentation as a bridge between textual and visual modalities to exploit semantic consistency.
• Developed GText-IF with multiple decoders and a degradation prompt module for adaptive mitigation of degradations.

Main Results:

• GText-IF demonstrated effective degradation-aware fusion in real-world degraded image scenarios.
• The framework exhibited remarkable advancement in fused image quality and information representation.
• Experiments confirmed the robustness and practicality of the proposed GText-IF approach.

Conclusions:

• GText-IF successfully addresses the challenge of degradation-aware image fusion by integrating text-driven semantics.
• The synergy between text, semantic segmentation, and a degradation prompt module enables adaptive mitigation of image degradations.
• The proposed framework offers a significant improvement over existing methods for fusing degraded images.