When our measurements are different every day: an ML-SEM simulation study on within-person nonuniform measurement...

This simulation study evaluated how model fit in multilevel structural equation models (ML-SEM) is affected by within-person nonuniform measurement bias in intensive longitudinal data (ILD). This kind of bias would be given if item discrimination (i.e., their factor loadings) in multiple-item questionnaires varied within person across time. Prior simulation studies and ILD studies tend to assume no such within-person bias, while such a bias implies that relations within measurement points are not comparable across time.

We simulating ILD under 450 conditions with various sample sizes, retesting frequencies, ICCs, and introduced within-person nonuniform measurement bias. We then investigated model (mis)fit in ML-SEM.

Type I error was well below nominal level. The χ² statistic and CFI outperformed the other fit indices (RMSEA, SRMR-w, SRMR-b), with the effects being conditional on all design factors. Furthermore, even though standard software solutions readily provide SRMR specific for each level, results discourage their use due to poor and non-stable performance.

While ecological momentary assessment motivated our study, our findings are applicable to other research settings that yield data with the same generally hierarchical structure (e.g., ambulatory assessment, daily diary studies, experience sampling methods). We conclude with practical recommendations on samples sizes and re-testing frequencies to offer guidelines for evaluating ML-SEM in ILD.