Reliability-Based Feature Weighting for Automated Essay Scoring.

Area of Science:

• Natural Language Processing
• Educational Measurement
• Artificial Intelligence in Education

Background:

• Traditional automated essay scoring (AES) systems prioritize emulating human scores, using statistical methods to determine feature importance.
• This reliance on human score prediction may lead to feature weights reflecting statistical artifacts rather than genuine writing quality.
• Machine essay evaluation differs fundamentally from human evaluation, suggesting a need for alternative scoring approaches.

Purpose of the Study:

• To propose and evaluate alternative feature weighting schemes for AES systems.
• To maximize the reliability and internal consistency of composite essay scores.
• To investigate if these alternative schemes outperform traditional human-prediction based weights.

Main Methods:

• Developed feature weighting schemes focused on optimizing score reliability and internal consistency.
• Compared these novel weighting schemes against traditional human-prediction based weights.
• Utilized a large-scale writing assessment dataset for empirical evaluation.

Main Results:

• Alternative feature weighting schemes produced significantly different feature weights compared to human-prediction methods.
• The proposed schemes resulted in comparable or superior reliability coefficients.
• Validity coefficients were also comparable or improved using the new weighting strategies.

Conclusions:

• Rethinking feature weighting in AES is crucial for developing more robust and reliable scoring systems.
• Prioritizing internal consistency over direct human score emulation offers a promising alternative for AES.
• This approach can lead to more effective automated assessment of writing skills.