Improving written-expression curriculum-based measurement feasibility with automated writing evaluation programs

Area of Science:

• Educational Technology
• Psychometrics
• Writing Assessment

Background:

• Automated Writing Evaluation (AWE) programs offer feasible alternatives for scoring student writing.
• Existing AWE tools require rigorous validation for accuracy, predictive validity, and potential bias.

Purpose of the Study:

• To evaluate the accuracy, predictive validity, and bias of automated scores for Written-Expression Curriculum-Based Measurement (WE-CBM).
• To compare automated WE-CBM metrics with traditional hand-scored metrics.

Main Methods:

• Collected writing samples from 722 students in Grades 2-5 using 3-minute WE-CBM tasks.
• Hand-scored four WE-CBM metrics and generated automated scores for the same metrics using a computer-based approach.
• Compared automated scores with hand-scored metrics and analyzed prediction of state-mandated writing test performance.

Main Results:

• Simpler automated metrics (total words, words spelled correctly) closely matched hand-calculated scores.
• Small discrepancies were found for more complex metrics (correct word sequences).
• Automated scores accurately predicted state test performance, similar to hand-scored metrics, with no identified bias against African American and Hispanic students.

Conclusions:

• Automated WE-CBM scoring demonstrates high accuracy and predictive validity, comparable to traditional methods.
• Automated scoring shows no evidence of bias for minority student groups.
• Findings support the use of automated scores for educational decision-making in writing assessment.