Differentiation of Illusory and True Halo in Writing Scores

This report summarizes an empirical study that addresses two related topics within the context of writing assessment-illusory halo and how much unique information is provided by multiple analytic scores. Specifically, we address the issue of whether unique information is provided by analytic scores assigned to student writing, beyond what is depicted by holistic scores, and to what degree multiple analytic scores assigned by a single rater display evidence of illusory halo. To that end, we analyze student responses to an expository writing prompt that were scored by six groups of raters-four groups assigned single analytic scores, one group assigned multiple analytic scores, and one group assigned holistic scores-using structural equation modeling. Our results suggest that there is evidence of illusory halo when raters assign multiple analytic scores to a single student response and that, at best, only two factors seem to be distinguishable in analytic writing scores assigned to expository essays.