Question evaluation for real-time surveys: Lessons from COVID-19 data collection

Area of Science:

• Public Health
• Survey Methodology
• Health Surveillance

Background:

• The COVID-19 pandemic highlighted the critical need for real-time, high-quality data for policy and public health.
• Novel public health events present unique challenges for survey data collection and question development.

Purpose of the Study:

• To describe an iterative, mixed-method approach for designing and evaluating survey questions during the COVID-19 pandemic.
• To identify and address specific challenges in survey question design for novel, rapidly evolving public health events.

Main Methods:

• A two-component, mixed-method approach was employed by the National Center for Health Statistics.
• Component one involved virtual cognitive interviewing to understand survey response processes.
• Component two utilized a two-round, mixed-method survey with a statistically-sampled panel for further development and subgroup analysis.

Main Results:

• Key challenges identified include a lack of shared understanding of novel concepts, shifting respondent reference periods, and the pervasive framing effect of the event.
• The iterative approach facilitated continuous improvement of COVID-19 survey items.
• Embedded probes (open- and close-ended) were developed using cognitive interview findings and initial survey results.

Conclusions:

• An iterative, mixed-method approach is effective for developing high-quality survey questions in novel and rapidly evolving public health contexts.
• This methodology provides a foundation for real-time data collection during future public health emergencies.
• Addressing challenges related to novel concepts and respondent perception is crucial for accurate health surveillance data.