Fixed or random testlet effects: a comparison of two multilevel testlet models

This simulation study compared the performance of two multilevel measurement testlet (MMMT) models: Beretvas and Walker's (2012) two-level MMMT model and Jiao, Wang, and Kamata's (2005) three-level model. Several conditions were manipulated (including testlet length, sample size, and the pattern of the testlet effects) to assess the impact on estimation of fixed and random effect parameters. The results of the present simulation study showed that MMMT-2r yielded the best parameter bias in estimation on fixed item effects, fixed testlet effects, and random testlet effects for conditions with nonzero equal pattern of random testlet effects' variance even when the MMMT-2r was not the generating model. However, random effects estimation did not perform well when unequal random testlet effects' variances were generated. Fit indices did not perform well either as other studies have had found. In addition, from the modeling perspective, MMMT-2r allows the greatest flexibility in terms of modeling testlet effects as fixed, random, or both.