The differential impact of resolution methods on the operational scores of gender and ethnic groups

In the scoring of performance assessments, when two raters assign different ratings some method must be used to resolve the discrepant ratings to form an operational score for reporting. This study investigated the differential impact of various resolution methods on the operational scores for gender and ethnic groups. The mean operational scores, and the passing rates for each group on two essay prompts were compared using three resolution methods (rater mean, tertium quid, and parity). The results indicated that for female and African American students, resolution typically resulted in greater reduction of mean operational ratings and passing rates than for those of their male or White counterparts. Differential item functioning (DIF) analyses were conducted using IRT-based logistic regression models. No apparent gender-related DIF was detected. Although uniform DIF was found for the ethnic groups, the effect was small, and there was not enough evidence to support the hypothesis that DIF could be associated with a resolution method.