Comparing Different Approaches of (Not) Accounting for Rapid Guessing in Plausible Values Estimation

Area of Science:

• Educational measurement
• Psychometrics
• Large-scale assessment

Background:

• Large-scale assessments inform educational policy but can be influenced by non-ability factors like motivation.
• Rapid guessing, a form of reduced motivation, can distort ability estimates and group comparisons.
• Current methods for addressing rapid guessing primarily focus on point estimates, neglecting latent variable research.

Purpose of the Study:

• To quantify the bias rapid guessing introduces into group comparisons using plausible value estimates.
• To introduce and evaluate novel methods for handling rapid guessing during plausible value estimation.
• To compare the performance of different models in accounting for rapid guessing.

Main Methods:

• A simulation study comparing four models: baseline (no correction), person-level correction, response-level correction (item response times), and a combined approach.
• Analysis of bias in group comparisons under different rapid guessing scenarios.
• Empirical validation using data from a German large-scale assessment (N = 478).

Main Results:

• The response-level and combined models demonstrated superior performance in accounting for rapid guessing.
• Correction at the person level alone was insufficient to mitigate the distorting effects of rapid guessing.
• The study successfully demonstrated the practical applicability of all evaluated approaches.

Conclusions:

• Response-level and combined modeling strategies are recommended for accurately estimating abilities in the presence of rapid guessing.
• Future research should focus on refining these methods to further enhance ability estimation in large-scale assessments.
• Accurate ability estimation is crucial for fair and effective educational policy and decision-making.