Know when to trust: Making AI scoring more reliable for educational assessment

Area of Science:

• Artificial Intelligence
• Educational Measurement
• Psychometrics

Background:

• Large language models (LLMs) offer new possibilities for automated scoring in education.
• Existing LLM scoring tools require enhancements for improved trustworthiness and accuracy.

Purpose of the Study:

• To introduce and evaluate three novel improvements for LLM-based automated scoring tools.
• To enhance the reliability and validity of LLM-generated scores in educational contexts.

Main Methods:

• Utilized data from over 20,000 responses to the Alternative Uses Test from 2,000+ participants.
• Implemented and assessed three techniques: model self-confidence, weighted probabilistic scoring, and ensemble modeling.
• Evaluated improvements by comparing LLM scores with human judges' assessments.

Main Results:

• All three introduced techniques demonstrated statistically significant positive results.
• Improvements were observed in correlation with human judges (from r=0.781 to r=0.823) and error reduction.
• The methods proved compatible as drop-in improvements for existing scoring techniques.

Conclusions:

• Model self-confidence, weighted probabilistic scoring, and ensemble modeling enhance LLM-based automated scoring.
• These adjustments boost the dependability and applicability of LLMs for quantitative text analysis in education.
• The findings support the use of LLMs for more trustworthy educational scoring.