Question evaluation for real-time surveys: Lessons from COVID-19 data collection

The need for high-quality, real-time data has never presented itself as clearly as it did during the COVID-19 pandemic. Responding to the COVID-19 pandemic, from both a policy and a public health perspective, required timely, accurate data about the public's attitudes and behaviors from health surveillance, monitoring, and public opinion surveys. The uniqueness of the COVID-19 pandemic also created particular challenges for survey data collection, specifically, how to develop high quality survey questions on topics that had never been previously fielded. To account for this challenge, the National Center for Health Statistics adopted an iterative, two-component, mixed-method approach to question design and evaluation. The first, a cognitive interviewing study using virtual, online interviews was used to produce interpretative schemata of the response processes underlying the survey questions. The second, a two-round, mixed method survey using a statistically-sampled panel, was designed to further develop the interpretive schemata and to allow for detailed subgroup analyses. To increase the usefulness of the survey's second round, cognitive interview findings and results from the survey's first round were used to develop both open- and close-ended embedded probes. Taken together, the studies reveal the specific problems for question-design during such a novel, quickly-evolving event: 1) a lack of shared understanding of novel concepts and vocabulary, 2) the shifting reference period respondents use to think about attitudes and behaviors during a multi-year event, 3) the pervasive nature of the event that therefore frames how respondents conceptualize and process questions about unrelated topics. This iterative approach to understanding question-design problems not only allowed for the continuing improvement of COVID-19 survey items, going forward, it also provided a methodological foundation for question development for high quality, real-time data collection.