Disentangling Selection Into Mode From Mode Effects

We investigate the impact of data collection mode on responses to variables in the National Social Life, Health, and Aging Project (NSHAP) Round 4 and discuss how potential mode differences should (and should not) be addressed in substantive analyses.

Among the set of respondents who were eligible to be contacted remotely in Round 4, we randomly selected 398 to be contacted instead for an in-person interview. We compare response rates and the distributions of selected key outcomes among those 398 respondents to those among the control group who were initially approached remotely. In contrast, we compared all R4 respondents according to the mode in which they completed the interview, including those not part of the randomized experiment.

Among those included in the experiment, there was no evidence of systematic differences in responses to physical and mental health questions between remote and in-person modes, nor in responses to number recall measures. In-person respondents scored moderately lower on cognitive function measures requiring careful attention to a figure and/or task, though this difference became less with each similar item. Remote respondents named fewer social network members. Comparing all respondents according to their final mode yielded substantially different results in all cases.

Mode did not appear to affect reports of physical and mental health based on a randomized comparison, though it did moderately affect other items in predictable ways. Naïve estimates of mode effects based on comparing all respondents according to mode yielded misleading results, and should not be used to adjust for mode differences in analyses.