Applying psychometrics to large language model simulated populations: recreating the HEXACO personality inventory...

Generative agents powered by large language models (LLMs) demonstrate human-like characteristics through sophisticated natural language interactions. Their ability to assume roles and personalities based on predefined character biographies has positioned them as cost-effective substitutes for human participants in social science research. This paper explores the validity of such persona-based agents in representing human populations; we recreate the HEXACO personality inventory experiment by surveying 310 GPT-4 powered agents, conducting factor analysis on their responses, and comparing these results to the original findings presented by Ashton et al. (Ashton et al. 2004 J. Pers. Soc. Psychol. 87, 707-721 (doi:10.1037/0022-3514.87.5.707). Our results found (i) a coherent and reliable personality structure was recoverable from the agents' responses demonstrating partial alignment to the HEXACO framework, (ii) the derived personality dimensions were consistent and reliable within GPT-4, when coupled with a sufficiently curated population, and (iii) cross-model analysis revealed variability in personality profiling, suggesting model-specific biases and limitations. We discuss the practical considerations and challenges encountered during the experiment. This study contributes to the ongoing discourse on the potential benefits and limitations of using generative agents in social science research and provides useful guidance on designing consistent and representative agent personas to maximize coverage and representation of human personality traits, while highlighting potential limitations of applying human psychometric frameworks to generative agents.