Bias of Two-Level Scalability Coefficients and Their Standard Errors

Area of Science:

• Psychometrics
• Statistical analysis
• Measurement theory

Background:

• Two-level Mokken scale analysis extends Mokken scale analysis to accommodate multi-rater data.
• Evaluating the accuracy of statistical methods in psychometrics is crucial for reliable data interpretation.

Purpose of the Study:

• To investigate the bias in estimated scalability coefficients and standard errors for two-level Mokken scale analysis.
• To assess the coverage of confidence intervals under various conditions.
• To compare the performance of the delta method and cluster bootstrap for standard error estimation.

Main Methods:

• Simulation studies were conducted under diverse testing conditions.
• Bias and coverage of scalability coefficients and standard errors were analyzed.
• The delta method and cluster bootstrap were compared for standard error estimation.

Main Results:

• Estimated scalability coefficients were unbiased across all tested conditions.
• The delta method provided accurate standard error estimates with good confidence interval coverage, except in specific cases (unequal raters, small item sets).
• The cluster bootstrap systematically underestimated standard errors, leading to low coverage.

Conclusions:

• The delta method is a reliable approach for estimating standard errors in two-level Mokken scale analysis.
• The cluster bootstrap method requires adaptation for accurate standard error estimation in this context.
• Using the harmonic mean in the delta method can improve standard error estimates when raters per subject are unequal.