Revisiting reliability with human and machine learning raters under scoring design and rater configuration in the...

Area of Science:

• Educational measurement
• Psychometrics
• Artificial intelligence in education

Background:

• Constructed-response (CR) items assess higher-order skills but face challenges with human scoring variability and cost.
• Machine learning (ML) offers a scalable solution, but its psychometric implications in rater-mediated models require further investigation.

Purpose of the Study:

• To examine the impact of scoring design, rater bias, ML inconsistency, and model specification on ability estimation reliability in polytomous CR assessments.
• To compare the performance of different estimation models under various scoring conditions.

Main Methods:

• Monte Carlo simulation was used to manipulate human and ML rater bias, ML inconsistency, and scoring density (complete, overlapping, isolated).
• Five estimation models were compared, including the Partial Credit Model (PCM) and the Many-Facet Partial Credit Model (MFPCM).

Main Results:

• Systematic bias, rather than random inconsistency, was identified as the primary source of error in ability estimation.
• Hybrid human-ML scoring improved estimation with unbiased or opposing biases but exacerbated errors when biases aligned.
• The Partial Credit Model (PCM) with fixed thresholds consistently outperformed more complex models, and anchoring CR items stabilized MFPCM estimation.

Conclusions:

• Scoring design and bias structure, not model complexity, determine the effectiveness of hybrid scoring.
• Anchoring constructed-response items to selected-response metrics is a practical strategy for enhancing estimation stability in ML-assisted scoring.