Contrast sensitivity in multimodal large language models: A psychophysics-inspired evaluation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Cognitive Science

Background:

• Evaluating the perceptual capabilities of Multimodal Large Language Models (MLLMs) requires understanding their processing of low-level visual features.
• Current methods for assessing artificial network vision often rely on internal model activations or classifier proxies, lacking a direct behavioral link.
• Human psychophysics offers a framework for understanding visual perception through behavioral responses to stimuli.

Purpose of the Study:

• To introduce a novel psychophysical method for estimating the Contrast Sensitivity Function (CSF) in MLLMs.
• To systematically characterize MLLMs' visual frequency tuning and compare it to human perception.
• To establish a scalable diagnostic tool for assessing multimodal perception in AI.

Main Methods:

• Treated MLLMs as end-to-end observers by querying them with structured prompts while presenting noise-based stimuli filtered at various spatial frequencies.
• Derived psychometric functions from the models' binary verbal responses to estimate contrast thresholds.
• Calculated the Contrast Sensitivity Function (CSF) for each MLLM without using internal activations.

Main Results:

• Some MLLMs exhibited CSFs with shapes or scales similar to humans, but none matched both.
• CSF estimates were highly sensitive to prompt phrasing, indicating limited linguistic robustness in MLLMs.
• Estimated CSFs effectively predicted MLLM performance under frequency-filtered and adversarial visual conditions.

Conclusions:

• Significant systematic differences exist in visual frequency tuning across various MLLMs.
• The proposed CSF estimation method provides a scalable and direct behavioral diagnostic tool for multimodal perception.
• Further research is needed to improve the linguistic robustness and perceptual accuracy of MLLMs.