Ordinal models to analyze strategy sophistication: Evidence from a learning trajectory efficacy study

Investigators often rely on the proportion of correct responses in an assessment when describing the impact of early mathematics interventions on child outcomes. Here, we propose a shift in focus to the relative sophistication of problem-solving strategies and offer methodological guidance to researchers interested in working with strategies. We leverage data from a randomized teaching experiment with a kindergarten sample whose details are outlined in Clements et al. (2020). First, we describe our problem-solving strategy data, including how strategies were coded in ways that are amenable to analysis. Second, we explore what kinds of ordinal statistical models best fit the nature of arithmetic strategies, describe what each model implies about problem-solving behavior, and how to interpret model parameters. Third, we discuss the effect of "treatment", operationalized as instruction aligned with an arithmetic Learning Trajectory (LT). We show that arithmetic strategy development is best described as a sequential stepwise process and that children who receive LT instruction use more sophisticated strategies at post-assessment, relative to their peers in a teach-to-target skill condition. We introduce latent strategy sophistication as an analogous metric to traditional Rasch factor scores and demonstrate a moderate correlation them (r = 0.58). Our work suggests strategy sophistication carries information that is unique from, but complimentary to traditional correctness-based Rasch scores, motivating its expanded use in intervention studies.