A taxonomy of reliability indices for cluster-level constructs measured by multiple individual-level indicators

Generalizability theory (GT) provides a variety of intraclass correlation coefficients (ICCs) to quantify reliability across different conditions of measurement-for example, scale and interrater reliability quantify generalization across items and raters, respectively. In a multilevel modelling (MLM) context, an ICC is often used to quantify the proportion of a subject-level variable's variance that is attributable to cluster-level differences (e.g., between schools or neighbourhoods). When data from cluster members are their ratings of a group-level characteristic, the ICC is an inter-rater reliability coefficient, interpreted as the degree to which observations of a cluster generalize across members of that cluster. Structural equation modelling (SEM) is becoming a popular framework for modelling GT, and the 'omega' indices of composite reliability-which are functions of SEM parameters-are special cases of GT (i.e., how measurements of a construct generalize across indicators). The merger of MLM and SEM (i.e., ML-SEM) provides a unique opportunity to explore how well measurements of cluster-level constructs generalize across multiple measurement conditions (indicators, cluster members, or both). A taxonomy of ICCs is explicated to appropriately quantify reliability across aspects of a procedure for measuring cluster-level constructs, adding value to the limited ICCs currently considered in ML-SEM applications.