Linking essay-writing tests using many-facet models and neural automated essay scoring

Area of Science:

• Educational Measurement
• Psychometrics
• Artificial Intelligence in Education

Background:

• Essay scoring is challenged by rater variability (severity, consistency).
• Item Response Theory (IRT) models, like many-facet Rasch models, address rater effects in ability estimation.
• Test linking is crucial for unifying scales when different examinee-rater groups exist, but traditional methods are often impractical.

Purpose of the Study:

• To develop a novel method for linking IRT models with rater parameters.
• To overcome the impracticalities of traditional test-linking methods in real-world scenarios.
• To accurately unify measurement scales across distinct examinee-rater groups.

Main Methods:

• Utilized neural automated essay scoring technology.
• Developed a new approach for linking IRT models and rater parameters.
• Compared the novel method with traditional linear linking.

Main Results:

• The proposed method successfully accomplished test linking.
• The accuracy of the novel method was comparable to linear linking.
• Effective linking was achieved even with few common examinees.

Conclusions:

• Neural automated essay scoring offers a viable solution for complex test linking in educational measurement.
• The new method enhances the accuracy and practicality of ability estimation in essay tests.
• This approach facilitates more reliable score comparisons across different rater groups.