Rasch Versus Classical Equating in the Context of Small Sample Sizes.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Equating and scaling are crucial for small-sample exams, particularly in specialized professions.
• Classical and Rasch-based methods have been proposed for addressing these challenges.
• Previous research highlights the need for robust equating techniques with limited data.

Purpose of the Study:

• To directly compare classical and Rasch equating techniques for small sample sizes (N≤100).
• To investigate the potential of pooling data across multiple exam forms within the Rasch framework to enhance estimation accuracy.

Main Methods:

• Simulation study using real data from a larger certification exam program.
• Resampling techniques to create multiple years of small-sample exam data (N≤100 per form).
• Comparison of WINSTEPS-based Rasch methods and Bayesian Markov Chain Monte Carlo (MCMC) methods.

Main Results:

• Pooling data across multiple administrations using Rasch models significantly improved equating accuracy compared to classical methods.
• WINSTEPS-based Rasch methods utilizing pooled data outperformed Bayesian MCMC methods.
• Prior distributions in MCMC methods introduced bias in score prediction when exam forms had differing item difficulties.

Conclusions:

• Rasch modeling, particularly when pooling data from multiple exam forms, offers a superior approach to equating for small-sample examinations.
• The WINSTEPS software provides a reliable Rasch-based solution for small-sample equating, outperforming MCMC methods in this context.
• Careful consideration of prior distributions is necessary when employing Bayesian methods for equating small-sample exams with potential form differences.