Language-Driven Cross-Attention for Visible-Infrared Image Fusion Using CLIP

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Single-modality imaging struggles in low-light or complex environments.
• Multimodal fusion (visible and infrared) is crucial for robot perception.
• Enhanced scene understanding aids navigation, localization, and environmental perception.

Purpose of the Study:

• To develop a text-guided infrared and visible image fusion network.
• To improve multimodal fusion effectiveness by integrating semantic information.
• To generate high-quality fused images with enhanced visual detail and semantic understanding.

Main Methods:

• A novel framework combining an image fusion branch and a text-guided module.
• Cross-domain attention mechanism for merging multimodal features.
• CLIP model for extracting semantic cues from text descriptions to guide fusion.

Main Results:

• Achieved strong quantitative performance on LLVIP and TNO benchmark datasets.
• Demonstrated robust and scalable performance in multimodal perception tasks.
• Generated high-quality color-fused images that enrich semantic understanding.

Conclusions:

• The proposed text-guided fusion network effectively integrates visual and linguistic information.
• The framework offers a promising solution for real-world multimodal perception applications.
• Enhanced fusion capabilities are critical for autonomous systems in complex environments.