Using classical and modern measurement theories to explore rater, domain, and gender influences on student writing...

This study i) examined the rater, domain, and gender influences on the assessed quality of student's writing ability and ii) described and compared different approaches for examining these influences based on classical and modern measurement theories. Twenty raters were randomly selected from a group of 87 trained raters contracted to rate essays of the annual Georgia High School Writing Test. Each rater scored the entire set of 375 essays on a 1-4 rating scale (366 essays were used in the analyses because nine cases had missing values and were dropped). Two approaches, the classical approach and the item response theory-based Rasch model, were used to conduct psychometric measures of reliability and inter-rater reliability, and statistical analyses with rater and gender as the predictor variables and the total and domain scores as the dependent variables. To achieve the second purpose, the Classical Test Model and the Rasch model were compared and contrasted and their strengths and limitations discussed as they related to student writing assessment. Analyses from both approaches indicated statistically significant rater and gender effects on student writing. Using domain scores as the dependent variables, there was a statistically significant rater by gender interaction effect at the multivariate level, but not at the univariate level. The Rasch analysis indicated a statistically significant rater by gender effect. The comparison between the two approaches highlighted their strengths and limitations, their different measurement and statistical models, and their different procedures.