The consequences of modeling autocorrelation when synthesizing single-case studies using a three-level model

Area of Science:

• Behavioral Science
• Psychology
• Educational Research

Background:

• Single-case study (SCS) data synthesis is increasingly common.
• Three-level models are frequently used to account for nested data structures (observations within participants within studies).
• The assumption of first-order autoregressive (AR(1)) errors at Level-1 is often made, but its validity is not always certain.

Purpose of the Study:

• To evaluate the performance of three-level models for SCS meta-analysis.
• To investigate the impact of misspecified Level-1 error structures (AR(1) vs. independent vs. moving-average) on model results.
• To examine how varying simulation conditions (series length, participants per study, studies per meta-analysis, variance components) affect model accuracy.

Main Methods:

• Monte Carlo simulation methods were employed.
• Three-level hierarchical linear models were simulated under various error covariance structures.
• Key parameters assessed included fixed effects (e.g., treatment effects) and variance components at Levels 1, 2, and 3.

Main Results:

• Fixed effects, such as average treatment effects and trends, were generally unbiased across conditions.
• Confidence intervals for fixed effects demonstrated good accuracy, even with AR(1) overspecification or misspecification.
• Variance components, especially at Level-2 (participants within studies) and Level-3 (studies), exhibited significant bias.

Conclusions:

• Three-level models can provide unbiased fixed-effect estimates in SCS meta-analysis, even with imperfect Level-1 error assumptions.
• Researchers should exercise caution when interpreting Level-2 and Level-3 variance components due to potential bias.
• Further research may be needed to develop robust methods for estimating variance components in SCS meta-analysis.