Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework

The purpose of this study is to examine the reliability and dependability of difference scores computed as the change between a pretest and a posttest administered to assess the effectiveness of an intervention. The data-collection design involved a nested structure, with persons (p) nested within groups (g), and groups nested within sites (s). Multivariate generalizability theory was employed to estimate the reliability and dependability of difference scores at the levels of persons, groups, and sites. The G study designs included $(p^{•}:g^{•}:s^{•})x{i}^{•}$ , $(p^{•}:s^{•})x{i}^{•}$ , $(p^{•}:g^{•})x{i}^{•}$ , $g^{•}x{i}^{•}$ , $p^{•}x{i}^{•}$ , and $s^{•}x{i}^{•}$ , with pretest and posttest serving as the two levels of the multivariate facet. In the designs with sites as the object of measurement, $(P^{•}:G^{•}:s^{•})x{I}^{•}$ , $(P^{•}:s^{•})x{I}^{•}$ , and $s^{•}x{I}^{•}$ , omitting groups within sites or persons within groups led to an underestimation of error variances and inflated generalizability and dependability coefficients. The relative error correlations increased, and the absolute error and universe score correlations decreased across $s^{•}x{I}^{•}$ , $(P^{•}:s^{•})x{I}^{•}$ , and $(P^{•}:G^{•}:s^{•})x{I}^{•}$ . Across all designs, generalizability and dependability coefficients were similar in magnitude, primarily due to the relatively small variance in items. Compared across different objects of measurement, both the generalizability and dependability coefficients were highest when the object of measurement was persons, and lowest when it was sites.