Psychometric Evaluation of Large Language Model Embeddings for Personality Trait Prediction

Area of Science:

• Computational Linguistics
• Psychometrics
• Artificial Intelligence

Background:

• Large language models (LLMs) show promise for assessing psychological constructs like personality traits.
• Prior research has focused on zero-shot or few-shot inference, with limited psychometric validation of LLM embeddings.
• The efficacy of LLM embeddings compared to traditional feature engineering for personality assessment is underexplored.

Purpose of the Study:

• To evaluate LLM embeddings for personality trait prediction.
• To compare LLM embeddings against zero-shot methods and traditional feature engineering.
• To assess the psychometric validity and correlations with linguistic/emotional markers of LLM embeddings.

Main Methods:

• Generated text embeddings from 1 million Reddit posts using RoBERTa, BERT, and OpenAI LLM architectures.
• Trained a bidirectional long short-term memory model for personality prediction and compared it with zero-shot inference.
• Assessed psychometric validity (reliability, convergent validity) and correlated embeddings with LIWC and emotion features.

Main Results:

• LLM embeddings significantly outperformed zero-shot approaches by 45% across personality traits.
• Moderate reliability (Cronbach α = 0.63) was observed, with strong correlations between embeddings and linguistic/emotional markers.
• LLM embeddings inherently captured key linguistic features, negating performance gains from advanced feature engineering.

Conclusions:

• LLM embeddings provide a robust, efficient alternative to zero-shot methods for personality trait analysis.
• Embeddings capture linguistic patterns effectively, reducing the need for extensive feature engineering.
• Future research should focus on fine-tuning strategies to enhance the psychometric validity of LLM embeddings.