A flexible behavioral method for measuring human and artificial intelligence alignment using representational...

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Evaluating large language models (LLMs) for societal roles necessitates understanding their alignment with human cognition.
• Existing methods lack comprehensive approaches to compare AI information representation with human understanding across tasks.

Purpose of the Study:

• To adapt and apply Representational Similarity Analysis (RSA) for quantifying AI-human alignment in information representation.
• To assess the cognitive alignment of various LLMs with human judgments across different modalities.

Main Methods:

• Utilized pairwise ratings within an RSA framework to measure AI-human representational similarity.
• Evaluated multiple LLMs, including GPT-5-mini, Claude Sonnet 4.5, and Llama-4, across text, sentence, and image data.

Main Results:

• GPT-5-mini and Claude Sonnet 4.5 demonstrated the highest alignment with human text ratings.
• Llama-4 emerged as the top-performing open-source model in terms of alignment.
• No LLM fully captured human inter-individual variability or achieved high alignment with individual human responses.

Conclusions:

• RSA provides a viable method for assessing LLM alignment with human cognition across diverse data types.
• While progress is evident, significant gaps remain in LLM's ability to replicate nuanced human cognitive variability.
• Further research is needed to enhance LLM's capacity to mirror human-like information processing and decision-making.