A priori reliability of tests with cut score

Area of Science:

• Psychometrics
• Educational Measurement
• Item Response Theory

Background:

• Mastery testing aims to classify examinees based on proficiency.
• Accurate classification relies on understanding potential misclassification errors.
• The Rasch model provides a framework for analyzing test performance.

Purpose of the Study:

• To theoretically compute the probability of misclassification in mastery tests.
• To define and compute ability intervals that characterize mastery test functioning.
• To investigate the impact of item difficulty on these intervals and explore extensions to the 2PL model.

Main Methods:

• Utilizing the raw score probability distribution within the Rasch model.
• Calculating misclassification probability as a function of latent ability.
• Analyzing the relationship between misclassification curves, ability distributions, and expected classification errors.

Main Results:

• Developed an exact computation for misclassification probability.
• Identified distinct ability thresholds influencing classification reliability.
• Defined two key ability intervals (near and far from the cut score) crucial for mastery test functioning.

Conclusions:

• The misclassification probability curve and latent ability distribution fully determine expected classification errors.
• Ability intervals provide insights into mastery test performance and are influenced by item difficulty.
• The methodology offers a robust approach to evaluating mastery test accuracy, with potential extensions to other models like the 2PL.