Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4

Area of Science:

• Educational Psychology
• Writing Assessment
• Educational Technology

Background:

• The Common Core State Standards emphasize writing proficiency, posing a risk for students with weaker writing skills on English language arts (ELA) assessments.
• Early identification of at-risk writers is crucial for targeted intervention and academic success.

Purpose of the Study:

• To evaluate the effectiveness of automated scoring for a Direct Assessment of Writing (DAW) as a universal screening tool for identifying third and fourth graders at risk of not meeting grade-level writing standards.
• To determine the diagnostic accuracy of DAW screening models in predicting performance on the summative Smarter Balanced ELA assessment.

Main Methods:

• A diverse sample of 100 third graders and 130 fourth graders completed Fall and Spring writing screeners using DAW procedures.
• Essays were scored using the Project Essay Grade (PEG) automated essay scoring system.
• Receiver Operating Characteristic (ROC) curve analysis and logistic regression were employed to assess screening model performance, calculating Area Under the Curve (AUC), sensitivity, and specificity.

Main Results:

• For third graders, AUC values for the DAW screeners were in the acceptable range (Fall=0.74, Spring=0.75).
• For fourth graders, AUC values approached or fell within the excellent range (Fall=0.79, Spring=0.83).
• Selected cutpoints demonstrated varying levels of sensitivity and specificity, indicating the models' ability to correctly identify students at risk and those meeting standards.

Conclusions:

• Automatically-scored DAW demonstrates significant potential as a valid and efficient universal screening tool for identifying writing difficulties in elementary students.
• This automated approach can support early intervention efforts, helping to ensure students meet grade-level writing expectations within the context of current educational standards.