Estimation of decision consistency indices for complex assessments: model based approaches

Area of Science:

• Educational Measurement
• Psychometrics
• Statistics

Background:

• The No Child Left Behind Act mandates proficiency level assessments, increasing the need for reliable classification consistency in educational testing.
• Existing methods for calculating classification consistency often rely on distributional assumptions or lack conditionality on specific student scores and cut scores.
• Recent methods, while distribution-free, fail to account for score location, leading to less precise consistency estimates, particularly for students near proficiency thresholds.

Purpose of the Study:

• To introduce and validate a novel conditional decision consistency statistic based on the Rasch measurement model.
• To provide a more accurate estimation of classification consistency that is conditional on individual student measures and cut score locations.
• To offer a flexible method adaptable for multiple classification categories and applicable to each examinee.

Main Methods:

• The proposed method utilizes the asymptotic standard error of measurement derived from the Rasch model.
• It calculates a conditional decision consistency statistic for each individual student's ability estimate (raw score).
• The method was evaluated using retest simulations on data fitting the Rasch model, comparing its performance against true score and bootstrap methods.

Main Results:

• The standard error method provides a conditional statistic, offering a likelihood of consistent classification upon retesting for each student.
• This approach yields more accurate estimates of classification consistency compared to true score and bootstrap methods, especially for students near cut scores.
• The method demonstrated superior performance in retest simulations, indicating enhanced reliability in classification decisions.

Conclusions:

• The conditional standard error method offers a significant improvement in estimating classification consistency for educational assessments.
• This approach provides a more nuanced and accurate understanding of testing reliability, particularly for students at critical proficiency levels.
• The proposed method is a valuable tool for improving the interpretation and application of assessment results in educational policy and practice.