Standard-Setting Procedures for Counts Data

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Setting accurate cut scores is crucial for educational assessment.
• Existing methods may not fully capture nuances in count data.
• The Rasch Poisson Counts Model (RPCM) offers a robust framework for analyzing count data.

Purpose of the Study:

• To propose and evaluate two novel procedures for setting cut scores for count data.
• To apply these methods within the framework of the Rasch Poisson Counts Model (RPCM).
• To illustrate the practical application of these procedures in a real-world educational setting.

Main Methods:

• An examinee-centered procedure, adapting Longford's (1996) approach, links contrasting-groups judgments to the RPCM ability scale via random logistic regression.
• An item-centered procedure involves judges specifying the minimum performance level (minimum correct responses or maximum errors) for a minimally competent student.
• Both methods utilize the RPCM to model the counts data and estimate the position of the minimally competent student on the ability scale.

Main Results:

• The study successfully illustrated both the examinee-centered and item-centered procedures.
• The application demonstrated the feasibility of using these RPCM-based methods for standard-setting.
• The methods provide a structured approach to defining performance standards based on count data.

Conclusions:

• The proposed examinee- and item-centered procedures offer effective approaches for setting cut scores with count data.
• These methods, grounded in the Rasch Poisson Counts Model, enhance the precision and interpretability of educational assessments.
• The study highlights the utility of these novel techniques in practical standard-setting scenarios, particularly in areas like mental arithmetic assessment.