Potential scoring and predictive bias in interim and summative writing assessments

Area of Science:

• Educational Psychology
• Assessment and Evaluation
• Writing Studies

Background:

• Interim and summative writing assessments inform instructional decisions.
• Potential for rater and score type bias in student writing evaluations is understudied.
• Understanding bias is crucial for equitable educational practices.

Purpose of the Study:

• To investigate rater bias in interim writing assessments.
• To compare teacher ratings versus researcher ratings (blinded).
• To examine score predictive validity across diverse student groups.

Main Methods:

• Analysis of interim writing assessments and state summative data (N=2,621, Grades 3-11).
• Evaluations by familiar teachers and blinded researchers using analytic rubrics.
• Comparison of scores and predictive validity across demographic groups (gender, English learners, free/reduced-price lunch, special education).

Main Results:

• Teachers scored students higher than blinded researchers.
• Female students scored higher than males; English learners, students eligible for free/reduced-price lunch, and students eligible for special education scored lower.
• Score disparities were smaller with researcher ratings.
• Interim scores predicted state outcomes similarly across groups, but some groups had lower overall state scores.

Conclusions:

• Teacher ratings may introduce bias, disproportionately affecting marginalized student groups.
• Masking student identities in writing assessments can mitigate scoring bias.
• Written composition sections of high-stakes tests may be less biased than multiple-choice sections for historically marginalized students.