On the Consistency of Automatic Scoring with Large Language Models.

Area of Science:

• Artificial Intelligence
• Educational Measurement
• Natural Language Processing

Background:

• Large language models (LLMs) demonstrate potential for automated scoring tasks.
• Inconsistency in LLM scoring can arise from model variations and training data differences.
• Understanding LLM scoring consistency is crucial for reliable automated assessment.

Purpose of the Study:

• To investigate intra-LLM and inter-LLM scoring consistency across five LLMs.
• To examine the impact of temperature settings on LLM scoring consistency.
• To evaluate the relationship between scoring consistency and accuracy.
• To propose and assess a voting strategy for improving LLM scoring.

Main Methods:

• Evaluated scoring consistency of five LLMs (Claude, DeepSeek, Gemini, GPT, Qwen).
• Assessed consistency under varying temperature settings.
• Utilized constructed-response items from science education and ASAP datasets.
• Implemented a majority voting strategy across LLMs.

Main Results:

• LLMs exhibited near-perfect intra-LLM consistency, irrespective of temperature.
• Inter-LLM consistency was moderate, higher for easier items.
• Intra-LLM consistency surpassed inter-LLM consistency.
• Intra-LLM consistency did not correlate with accuracy; inter-LLM consistency showed a positive correlation.
• Majority voting enhanced scoring accuracy by combining diverse LLM strengths.

Conclusions:

• LLMs offer high internal scoring reliability but varying external agreement.
• Inter-LLM consistency is a better predictor of scoring accuracy than intra-LLM consistency.
• Ensemble methods, like majority voting, can mitigate LLM scoring inconsistencies and improve accuracy in educational assessments.