Mode effects for Add Health in transition from in-person to mixed mode

As longitudinal panel surveys face escalating costs, many are transitioning from traditional in-person interviews to mixed-mode designs. While essential for fiscal sustainability, this shift creates a methodological challenge: distinguishing genuine longitudinal change from measurement effects introduced by the change in mode. This study evaluated the impact of this transition within the National Longitudinal Study of Adolescent to Adult Health (Add Health) as the cohort entered midlife.

Utilizing Wave VI data and a randomized split-sample design (in which the eligible sample was randomly divided between a new web-first mixed-mode protocol [Sample 1] and an in-person control replicating the legacy survey protocol [Sample 2]), the study estimated the differential mode effect, capturing cross-sectional differences between the two designs. Subsequently, a Hidden Markov Model (HMM) was applied across Waves III-VI to decompose these design effects into mode-specific measurement error and residual nonresponse components.

Concerns regarding systemic declines in data quality were largely unfounded; the mixed-mode design outperformed the traditional in-person design in measurement accuracy. However, significant mode effects were identified for some items, particularly sensitive health outcomes and stigmatized behaviors, driven by distinct combinations of measurement error and differential nonresponse.

Because these differences can inadvertently mimic or obscure actual aging trends, researchers should screen variables for mode sensitivity and incorporate appropriate adjustments, such as a mode or design indicator, inverse propensity weighting, or HMM-based decomposition, in longitudinal analyses, ensuring that "signals" of aging are not obscured by survey redesign.