Beyond Scales and Essays: Artificial Intelligence Chatbots as Personality Interviewers

AI chatbots offer an interactive method to assess personality through semi-structured interviews. By leveraging natural language processing and machine learning, these chatbots can engage users in conversational exchanges, providing a more nuanced and contextual evaluation of personality traits. This study compares the psychometric validity of AI chatbot-based personality interviews with conventional methods, including scale-based measures and short essay questionnaires. We also examine how user perceptions of the AI chatbot influence the validity of machine-inferred personality scores. Data was collected from 189 Prolific participants at two time points. Adopting the multitrait-multimethod (MTMM) analytical framework and based on the generalizability theory (G-theory), we found that machine-inferred scores from both the short essay questionnaire and the AI chatbot interview converge well with scale-based methods but show weaker discriminant validity. Chatbot-based scores demonstrated incremental validity beyond other methods (i.e., scale-based and short essay scores) for predicting subjective health and perceived job performance. Preliminary findings suggest that ease of concentration during the chatbot interview may influence the convergence between scale-based and chatbot-based machine-inferred scores. Overall, our results support the use of AI chatbots for personality assessment while highlighting the need for further research on how user perceptions affect the psychometric properties of AI-based assessments.