Large Sample Confidence Intervals for Item Response Theory Reliability Coefficients.

Area of Science:

• Psychometrics
• Statistical Modeling

Background:

• Item response theory (IRT) applications often report reliability estimates for ability or sum scores.
• However, the literature lacks analytical expressions for the standard errors of reliability coefficient estimators, hindering reporting of reliability variability.

Purpose of the Study:

• To derive asymptotic variances for IRT marginal and test reliability coefficient estimators.
• To construct confidence intervals for these reliability coefficients.

Main Methods:

• Derivation of asymptotic variances for dichotomous and polytomous IRT models.
• Assumption of asymptotically normally distributed item parameter estimators.
• Construction of confidence intervals using derived variances.

Main Results:

• Confidence intervals for test reliability coefficients demonstrate good coverage in finite samples across various settings (e.g., generalized partial credit model, three-parameter logistic model).
• The marginal reliability coefficient estimator shows finite sample bias, leading to confidence intervals not achieving nominal levels with small sample sizes.
• This bias diminishes as sample size increases.

Conclusions:

• The derived methods provide a means to quantify uncertainty in IRT reliability estimates.
• Test reliability confidence intervals are reliable in practice, even with moderate sample sizes.
• Marginal reliability confidence intervals require larger sample sizes to achieve nominal coverage due to initial bias.