The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study

Area of Science:

• Statistics
• Quantitative Psychology
• Educational Research

Background:

• Three-level modeling is a valid statistical method for synthesizing single-subject experimental studies.
• Combining single-subject data across studies often necessitates standardization due to differing measurement scales.
• Dividing the dependent variable by the residual standard deviation is one approach to data standardization.

Purpose of the Study:

• To evaluate the effectiveness of standardizing single-subject data using the residual standard deviation within a three-level modeling framework.
• To assess the accuracy of estimating fixed effects (treatment effects) and variance components (between- and within-subject variance) using Monte Carlo methods.

Main Methods:

• Utilized Monte Carlo simulations to evaluate a specific data standardization approach (dividing the dependent variable by the residual standard deviation).
• Examined the estimation of fixed effects, including immediate treatment effect and treatment effect on the time trend.
• Assessed the accuracy of variance component estimation, specifically between- and within-subject variance.

Main Results:

• Three-level synthesis of standardized single-subject data is appropriate for estimating treatment effects.
• Accurate treatment effect estimation is enhanced with a larger number of studies (≥30) and measurement occasions per subject (≥20).
• Homogeneous studies (small between-study variance) further improve treatment effect estimation accuracy.
• Estimates for variance components were found to be less accurate compared to treatment effect estimates.

Conclusions:

• Standardizing single-subject data via division by residual standard deviation is a viable approach for three-level meta-analysis of treatment effects.
• The method demonstrates robustness for fixed effect estimation under various realistic conditions, especially with substantial data.
• Researchers should be cautious when interpreting variance component estimates derived from this standardization method.