Diff-MEF: Cross-Modal Diffusion Framework With Text Prompts and Semantic Perception for Multi-Exposure Image Fusion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multi-exposure image fusion (MEF) faces challenges due to the lack of real-world ground truth (GT).
• Existing benchmarks use pseudo GT from algorithm ensembles, which have imperfections.
• Current MEF methods struggle with performance and robustness due to pseudo GT limitations and fixed mapping relationships.

Purpose of the Study:

• To propose a novel cross-modal diffusion framework, Diff-MEF, for improved multi-exposure image fusion.
• To address the limitations of pseudo GT and fixed mapping relationships in existing MEF methods.
• To enhance the performance and robustness of MEF using text prompts and semantic perception.

Main Methods:

• Reformulated MEF as a probabilistic estimation task using a conditional diffusion model.
• Incorporated semantic and exposure priors as text prompts and semantic perception.
• Utilized multi-modal prior embedding and optimization guidance, including segmentation masks and exposure/content-aware text prompts.
• Employed a text-segmentation refinement module and a semantic-level contrastive loss in CLIP's semantic space.

Main Results:

• Diff-MEF demonstrated superior fusion performance compared to state-of-the-art (SOTA) methods.
• The proposed framework showed enhanced robustness across diverse exposure scenarios.
• Experiments confirmed the effectiveness of Diff-MEF in mitigating degradations introduced by pseudo GT and fusion distortions.

Conclusions:

• Diff-MEF offers a robust and high-performing solution for multi-exposure image fusion.
• The synergistic use of text prompts and semantic perception effectively overcomes pseudo GT limitations.
• The framework provides a significant advancement in MEF technology, with code available for further research.