A Test Can Have Multiple Reliabilities.

Area of Science:

• Psychometrics
• Educational Measurement
• Psychological Statistics

Background:

• Coefficient alpha is a widely used measure of internal consistency in psychometric research.
• Existing interpretations of coefficient alpha, such as Lord and Novick's true score theory, face empirical limitations.
• Generalizability theory offers an alternative framework for interpreting coefficient alpha.

Purpose of the Study:

• To advocate for the generalizability theory interpretation of coefficient alpha.
• To compare the empirical basis of generalizability theory's true scores with alternative models.
• To delineate conditions under which the generalizability interpretation is most appropriate.

Main Methods:

• Conceptual analysis comparing different theoretical frameworks for coefficient alpha.
• Examination of the assumptions underlying domain sampling versus stochastic subject models.
• Discussion of latent variable models and their implications for reliability estimation.

Main Results:

• Coefficient alpha is presented as a consistent, albeit slightly biased, estimate of the generalizability coefficient in a random subjects x items design.
• The domain sampling true scores used in generalizability theory are argued to have a stronger empirical foundation.
• The generalizability interpretation is favored over alternative models unless a latent variable model with proven validity (e.g., McDonald's omega) is applicable.

Conclusions:

• The generalizability theory interpretation of coefficient alpha provides a more robust and empirically grounded approach to reliability estimation.
• This interpretation is particularly valuable in designs involving random sampling of both subjects and items.
• Alternative interpretations are conditionally defensible when specific latent variable models are met, especially those implying essential tau-equivalence.