A Monte Carlo Study of Confidence Interval Methods for Generalizability Coefficient.

Area of Science:

• Psychometrics
• Statistical modeling
• Educational measurement

Background:

• Computing confidence intervals for generalizability coefficients is challenging.
• Generalizability coefficients are vital for assessing score trustworthiness, especially with small sample sizes.
• Generalizability theory can be framed using linear mixed-effect models (LMMs).

Purpose of the Study:

• To evaluate four LMM-based methods for computing confidence intervals in generalizability theory.
• To assess the accuracy of these methods under various simulated conditions.
• To identify the most reliable method for constructing confidence intervals for generalizability coefficients.

Main Methods:

• Linear mixed-effect models (LMMs) were utilized to frame generalizability theory.
• Four LMM-based methods for confidence interval computation were examined.
• Simulations were conducted using different test score types (normal, dichotomous, polytomous) and designs (p×i×r, p×[i:r]).
• A bootstrap technique ('parametric methods with spherical random effects') was compared against other LMM-based methods and a model-based approach.

Main Results:

• The 'parametric methods with spherical random effects' bootstrap technique demonstrated superior accuracy compared to three other LMM-based methods.
• The chosen bootstrap technique showed robust performance when compared to a model-based approach in a second simulation study.
• Accuracy was evaluated across varying numbers of examinees, raters, and items.

Conclusions:

• The 'parametric methods with spherical random effects' bootstrap approach is recommended for computing confidence intervals in generalizability theory.
• Confidence intervals should consistently accompany point estimates of generalizability coefficients.
• Accurate confidence intervals enhance the trustworthiness and interpretation of generalizability coefficients in research.