Inferring Personality From Social Media Activity Using Large Language Models: Cross-Model Agreement, Temporal...

Area of Science:

• Computational Social Science
• Psychological Assessment
• Artificial Intelligence

Background:

• Large language models (LLMs) show potential for unobtrusive personality trait inference from digital data.
• The reliability and validity of LLM-based personality assessments require further investigation.

Purpose of the Study:

• To evaluate the reliability and validity of personality trait inferences made by LLMs from social media data.
• To compare the performance of Gemini 1.5 Pro and GPT-4o in personality assessment.

Main Methods:

• Inferred Big Five personality traits using Gemini 1.5 Pro and GPT-4o from 2 years of Facebook posts (N=1214 Italian users).
• Compared LLM predictions against self-reported personality using the Ten-Item Personality Inventory.

Main Results:

• LLM predictions showed variability, underestimating Agreeableness and Conscientiousness, overestimating Extraversion, while Neuroticism and Openness aligned with self-reports.
• Aggregating inferences across LLMs and time points enhanced reliability and temporal stability, with cross-LLM agreement reaching 0.83 for Extraversion.
• Correlations with self-reports were modest, with the highest being 0.31 for Openness when combining LLM inferences.

Conclusions:

• Aggregating LLM inferences across models and time points is crucial for improving the reliability and validity of personality assessments.
• LLMs offer a promising, albeit imperfect, tool for personality inference from digital footprints.