A rank-ordering method for equating tests by expert judgment

This paper describes a new method of comparing the raw mark scales on two tests using expert judgment. The two tests do not need to have any common items, nor to be taken by common groups of candidates. This study used scripts (i.e., the complete work of a candidate on the test) from England's National Curriculum Test for Reading at Key Stage 3 (14-year olds) in 2003 and 2004. Each member of a panel of 12 experts was given four packs each containing ten scripts--five scripts from each year's test. Marks and annotations from these scripts had been removed. Their task was to put the ten scripts into a single rank order, based on a holistic judgment of the level of performance exhibited in each. Because the design of the study linked scripts across judges and packs it was possible to construct a single latent trait of judged quality of performance. This was done using two different analytical methods: the Rasch formulation of Thurstone paired comparisons, and the Rasch Partial Credit model. Relating the two raw mark scales to the single latent scale allowed the two years' tests to be equated. The merits of using this standard-maintaining method as opposed to a standard-setting method in this particular context are discussed.