The Best of Two Worlds: IRT-Enhanced Automated Essay Interpretable Scoring

Area of Science:

• Artificial Intelligence in Education
• Educational Measurement
• Natural Language Processing

Background:

• Automated Essay Scoring (AES) systems face challenges with opaque decision-making and limited validation in diverse linguistic contexts.
• Educator trust in AES is hindered by the lack of transparency in scoring algorithms.
• Existing AES models often lack robust validation across different educational settings and languages.

Purpose of the Study:

• To propose IRT-AESF, an innovative framework integrating item response theory (IRT) with deep learning for transparent and interpretable automated essay scoring.
• To generate theoretically grounded psychometric parameters (student ability, item difficulty, item discrimination) for enhanced understanding of scoring decisions.
• To address the limitations of current AES systems by improving transparency and validation across diverse educational contexts.

Main Methods:

• Developed the IRT-AESF framework by integrating item response theory (IRT) principles with deep learning models.
• Generated three key psychometric parameters: student ability, item difficulty, and item discrimination.
• Evaluated the framework using 5-fold cross-validation on three large-scale datasets (41,328 essays) from English and Chinese educational settings.

Main Results:

• IRT-AESF demonstrated statistically significant improvements over baseline models.
• Achieved an 8.4% relative increase in quadratic weighted kappa, indicating enhanced scoring accuracy and reliability.
• Maintained robust cross-lingual performance, validating its effectiveness in diverse linguistic environments.

Conclusions:

• The IRT-AESF framework offers a transparent and interpretable approach to automated essay scoring.
• This research advances the development of trustworthy AI-powered assessment systems.
• The framework provides meaningful diagnostic insights beyond simple scores, supporting educational practice.