AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories.

Area of Science:

• Psychology
• Artificial Intelligence
• Computational Linguistics

Background:

• Large language models (LLMs) inadvertently acquire psychological traits from vast training text corpora.
• These acquired traits, reflecting human authors' personalities and biases, can influence LLM behavior and real-world consequences.
• Assessing these noncognitive traits is crucial for understanding LLM behavior.

Purpose of the Study:

• To demonstrate the repurposing of standard psychometric inventories as diagnostic tools for evaluating analogous traits in LLMs.
• To provide a method for bringing LLMs' acquired psychological traits to light.
• To enable psychometric profiling for comparative analysis of LLMs' noncognitive characteristics.

Main Methods:

• Utilizing language-based psychometric inventories to elicit responses from LLMs.
• Applying zero-shot classification as a promising psychometric approach for LLMs.
• Testing the approach across several LLMs and various psychometric inventories.

Main Results:

• Successfully repurposed psychometric inventories to evaluate analogous traits in LLMs.
• Demonstrated the feasibility of zero-shot classification for LLM psychometric profiling.
• Provided empirical evidence for LLMs exhibiting or mimicking human-like psychological traits.

Conclusions:

• Psychometric profiling offers a valuable window into the personalities, values, beliefs, and biases of LLMs.
• This methodology facilitates the study and comparison of LLMs based on noncognitive characteristics.
• Highlights open challenges and future research directions in the emerging field of AI Psychometrics.