Classification accuracy and efficiency of writing screening using automated essay scoring

Area of Science:

• Educational Technology
• Natural Language Processing
• Psychometrics

Background:

• Automated Essay Scoring (AES) offers potential for efficient and objective writing assessment.
• Traditional writing assessments can be time-consuming and subjective.
• Developing effective writing screeners is crucial for identifying students needing academic support.

Purpose of the Study:

• To evaluate the efficacy of an AES-based writing screener (Project Essay Grade - PEG) for identifying students at risk of failing state English language arts tests.
• To determine if a more efficient AES-screener with fewer prompts maintains classification accuracy.
• To compare the accuracy of AES-scored screeners against word count-based screeners.

Main Methods:

• Utilized Project Essay Grade (PEG) for automated essay scoring.
• Students in Grades 3-5 (n=185, 167, 187) wrote six essays across narrative, informative, and persuasive genres.
• Employed Receiver Operating Characteristic (ROC) curve analysis to assess classification accuracy, including sensitivity, specificity, and predictive probabilities.

Main Results:

• AES-scored screeners, both multi-prompt and fewer-prompt versions, demonstrated acceptable classification accuracy.
• AES-based screeners were found to be more accurate and efficient than screeners based solely on word count.
• The study confirmed the viability of using AES for writing screening.

Conclusions:

• Automated Essay Scoring (AES) provides a viable and efficient tool for writing screening.
• AES-based screeners accurately identify students at risk of academic failure in English language arts.
• The findings support the integration of AES technology into large-scale educational assessments.