A Causal Lens on Non-RGB Vision Sensor Understanding in Vision-Language Models.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Vision-Language Models (VLMs) excel with RGB images but fail with non-RGB sensor data (thermal, depth, hyperspectral, X-ray).
• This failure is due to an RGB-centric bias, causing VLMs to misinterpret unique physical properties of non-RGB modalities.

Purpose of the Study:

• To systematically evaluate and address the RGB-centric bias in VLMs using non-RGB sensor data.
• To introduce a novel benchmark suite, CausalSense, and a causal learning framework to mitigate this bias.

Main Methods:

• Developed CausalSense, a benchmark suite for evaluating VLM bias on non-RGB data.
• Designed a causal learning framework using confounder dictionaries and backdoor adjustments.
• Integrated sensor-specific knowledge into VLMs without extensive retraining.

Main Results:

• State-of-the-art VLMs show significant performance deficits with non-RGB sensor comprehension.
• The proposed causal deconfounded cross-modal encoder substantially improved VLM reasoning about physical attributes.
• A measurable reduction in the performance gap was achieved.

Conclusions:

• Current VLMs exhibit a critical RGB-centric bias limiting their use with diverse sensor data.
• The CausalSense benchmark and causal framework enable more resilient, sensor-aware VLMs.
• This research facilitates VLM interpretation of phenomena beyond the visible spectrum.