Know when to trust: Making AI scoring more reliable for educational assessment

The rapid rise of large language models (LLMs) has created new opportunities for educational measurement. This paper introduces and evaluates three improvements to LLM-based automated scoring tools: model self-confidence, weighted probabilistic scoring, and ensemble modeling. The study utilizes data from over 20,000 responses to the Alternative Uses Test, the most popular divergent thinking task, from more than 2,000 participants across multiple studies. Model self-confidence taps into LLMs' internal mechanisms to gauge their confidence in probability estimates, helping identify when machine-generated outputs are trustworthy. Weighted probabilistic scoring considers a broader range of completion possibilities in deriving a final score. The final technique, ensemble models, assesses the performance gains from combining multiple models. These methods, tested in divergent thinking response scoring, each show statistically significant positive results, with improvements in correlation with human judges (from r = 0.781 to r = 0.823) and reduction in error. All three techniques improve the performance and trustworthiness of automated scoring models, and are compatible as drop-in improvements to existing techniques. The findings suggest that these adjustments can boost the dependability and applicability of LLMs in educational scoring, specifically for systems that derive a quantitative measure from a text input.