Composite reliability of multilevel data: It's about observed scores and construct meanings

This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Preacher, and Zyphur (2014), denoted as ω̃^b and ω̃^w (and also α̃^b and α̃^w). One major limitation of those indices is that they are quantities for latent, unobserved level-specific composite scores, and are not suitable for observed composites at different levels. As illustrated using simulated data in this article, ω̃^b can drastically overestimate the true reliability of between-level composite scores (i.e., observed cluster means). Another limitation is that the development of those indices did not consider the recent conceptual development on construct meanings in multilevel modeling (Stapleton & Johnson, 2019; Stapleton, Yang, & Hancock, 2016). To address the second limitation, this article defines reliability indices (ω^2l, ω^b, ω^w, α^2l, α^b, α^w) for three types of multilevel observed composite scores measuring various multilevel constructs: individual, configural, shared, and within-cluster. The article also shows how researchers can obtain sample point and interval estimates using the derived formulas and the provided R and Mplus code. In addition, a large-scale national data set was used to illustrate the proposed methods for estimating reliability for the three types of multilevel composite scores, and practical recommendations on when different indices should be reported are provided. (PsycInfo Database Record (c) 2021 APA, all rights reserved).