Scoring employment interviews with large language models: Evaluation design components, validity investigations, and...

Recently, supervised machine learning models served as alternatives to humans for scoring open-ended assessments, including employment interviews. More recently, large language models (LLMs) emerged as alternative raters that do not require task-specific training data. However, interview evaluation design considerations have predominantly been conceptualized for human raters, and thus, it is unclear whether the psychometric benefits of interview evaluation best practices generalize to LLM raters. Additionally, LLM raters introduce novel evaluation design decisions that could be psychometrically consequential. We investigated the effects of several LLM rater evaluation practices-specifically, prompt design, model selection, hyperparameters, and number of interviewers-on the LLM interview scores' psychometric properties in two interview data sets. In the first, interviewee Big Five personality traits were evaluated (N = 954). In the second, interviewees' question responses were evaluated on the targeted construct on Behaviorally Anchored Rating Scales (N = 144). We then investigated the LLM scores' intrarater reliabilities, test-retest correlations, convergent, discriminant, and criterion evidence of validity, group differences, and measurement bias. We compared this evidence, when possible, to the same evidence for human raters and supervised machine learning models. The results suggest that ensembles of larger, newer LLMs using prompts with detailed construct information hold potential for scoring employment interviews with psychometric properties comparable to or superior to supervised machine learning models and single human raters. We detail the reasons that organizations may want to be cautious in adopting LLMs for scoring high-stakes open-ended assessments, but since organizations have already begun adopting them, we also offer best practice recommendations. (PsycInfo Database Record (c) 2026 APA, all rights reserved).