Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items

Area of Science:

• Educational Measurement
• Psychometrics
• Artificial Intelligence in Education

Background:

• Open-ended test items enhance construct validity but require costly manual scoring.
• Manual scoring limits the use of item data in adaptive testing.
• Automatic scoring using machine learning offers efficiency but introduces classification errors.

Purpose of the Study:

• To develop statistical models that integrate both manual and automatic scores for open-ended items.
• To account for classification errors inherent in automatic scoring within measurement models.
• To improve the accuracy of ability estimation in educational assessments.

Main Methods:

• Proposed two joint Item Response Theory (IRT) models incorporating both manual and automatic scores.
• Extended existing IRT models to include a component for automatic scoring errors.
• Evaluated models using data from the Programme for International Student Assessment (PISA) 2012 and simulated datasets.

Main Results:

• The proposed joint models effectively mitigate the impact of classification errors on ability estimation.
• Demonstrated improved accuracy compared to a baseline model that ignored automatic scoring errors.
• Validated the models' performance on real-world (PISA) and simulated data.

Conclusions:

• Joint modeling provides a robust approach to handling automatically scored open-ended items in educational testing.
• Accounting for classification errors is crucial for accurate ability estimation when using automatic scoring.
• These models enhance the validity and utility of open-ended items in adaptive and large-scale assessments.