A rank-ordering method for equating tests by expert judgment

Area of Science:

• Educational measurement
• Psychometrics
• Assessment methodology

Background:

• Comparing performance across different test administrations is crucial for educational assessment.
• Traditional equating methods often require common items or common person samples, which are not always feasible.
• Expert judgment offers a potential alternative for score comparison when direct links are absent.

Purpose of the Study:

• To present and validate a new method for comparing raw mark scales of two tests using expert judgment.
• To equate the National Curriculum Test for Reading at Key Stage 3 from 2003 and 2004.
• To explore the application of latent trait models in conjunction with expert judgments for test equating.

Main Methods:

• Utilized scripts from the National Curriculum Test for Reading (Key Stage 3, 14-year-olds) from 2003 and 2004.
• A panel of 12 experts rank-ordered scripts based on holistic performance, with marks and annotations removed.
• Employed the Rasch formulation of Thurstone paired comparisons and the Rasch Partial Credit model to establish a single latent trait of judged quality.

Main Results:

• Successfully constructed a single latent trait representing judged performance quality from expert rankings.
• Equated the raw mark scales of the 2003 and 2004 tests by relating them to the common latent scale.
• Demonstrated the feasibility of using expert judgment and latent trait modeling for test equating.

Conclusions:

• The proposed method provides a viable alternative for test equating, particularly when common items or candidates are unavailable.
• Expert judgment, when structured and analyzed appropriately, can yield reliable measures of performance quality for equating purposes.
• This standard-maintaining approach offers advantages in specific contexts compared to standard-setting methods.