On the factor structure of standardized educational achievement tests

This research analyzed the factor structure at both the item- and subtest-level of California's norm- and criterion-referenced standardized educational achievement tests (SEAT) used in that state's high-stakes educational accountability assessments. It was shown through full information factor analysis and multidimensional IRT models (e.g., TESTFACT and NOHARM) that, at the item-level, SEATs are invariably highly unidimensional (i.e., they appear to tap a unidimensional theta scale) even when items representing such diverse content areas such as English, science, mathematics, and history are analyzed simultaneously as a single measure. These item-level factors also accounted for a relatively small proportion (1/4 to 1/3) of the variance. It was also shown that, when these tests are analyzed using more reliable indicators such as subtests, a much richer factor structure emerged that accounted for a larger portion (about 2/3) of the total common variance. As expected, these factor structure configurations (and underlying dimensionality) were preserved across the item- and subtest-levels. However, the factors emerging from both the item- and subtest-level analyses were highly correlated and produced strong second-order and general factors. The meaning underlying these results was examined, along with their implications with respect to the assumptions underlying modern approaches to test calibration, scaling, and score interpretation.