LTOFusion: A Learning-to-Optimize Framework With Flow Matching for Unsupervised Image Fusion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Multimodal Image Fusion (MMIF) synthesizes information from diverse sources for comprehensive images.
• Current deep learning methods struggle with generalizable patterns for complex image-to-image mappings in MMIF.
• Existing approaches face challenges in extracting robust and adaptable features for varied fusion scenarios.

Purpose of the Study:

• To introduce LTOFusion, a novel learning-to-optimize framework for multimodal image fusion.
• To decouple complex fusion problems into manageable multistage subproblems using trajectory optimization.
• To enhance fusion performance and generalizability across diverse applications.

Main Methods:

• Formulating image fusion as a trajectory optimization problem, breaking it into multistage subproblems.
• Employing a restricted state transition function based on flow matching for prediction space compression.
• Utilizing a memory-replay strategy with intermediate fusion states to improve training diversity and model robustness.
• Implementing a hybrid loss function incorporating intensity, gradient, structure, and local normalized cross-correlation for detailed results.

Main Results:

• LTOFusion achieves state-of-the-art performance in multimodal image fusion tasks.
• The method demonstrates superior results across multiple downstream applications.
• The framework achieves high performance without requiring task-specific fine-tuning.

Conclusions:

• LTOFusion offers a robust and generalizable framework for multimodal image fusion.
• The learning-to-optimize approach effectively addresses limitations of direct image-to-image mapping methods.
• The proposed method advances the field by improving fusion quality and applicability.