Large language models predict human sensory judgments across six modalities

Area of Science:

• Cognitive Science
• Philosophy of Mind
• Artificial Intelligence
• Computational Linguistics

Background:

• A long-standing challenge in cognitive science and philosophy is understanding how much perceptual information can be derived solely from language.
• Investigating the relationship between linguistic representations and human perception is crucial for understanding cognition.

Purpose of the Study:

• To determine the extent to which large language models (LLMs) can recover perceptual information from text.
• To establish a lower bound on the perceptual information extractable from language using state-of-the-art LLMs.
• To explore the impact of cross-linguistic variations on perceptual representations within LLMs.

Main Methods:

• Elicited pairwise similarity judgments from GPT models across six diverse psychophysical datasets.
• Compared LLM-generated judgments with human data to assess representational accuracy.
• Applied LLMs to a multilingual color-naming task to analyze cross-linguistic perceptual variations.

Main Results:

• LLM judgments showed significant correlations with human data across all tested domains, successfully recovering known perceptual structures like the color wheel and pitch spiral.
• GPT-4, a model co-trained on vision and language, did not show modality-specific advantages, yielding similar results with or without direct visual input.
• LLMs replicated known cross-linguistic variations in color naming (e.g., English vs. Russian), demonstrating an understanding of language-perception interactions.

Conclusions:

• State-of-the-art LLMs can extract substantial perceptual information from language, serving as a valuable tool for cognitive science research.
• Perceptual representations in LLMs are robust, correlating highly with human judgments even when relying solely on textual descriptions.
• LLMs can model the influence of specific languages on perceptual categorization, offering insights into the interplay between language and perception.