Health estimate differences between six independent web surveys: different web surveys, different results?

Area of Science:

• Survey Methodology
• Public Health Research
• Data Science

Background:

• General population web surveys frequently utilize online panels from commercial agencies.
• Many of these panels rely on non-probability samples, raising concerns about data representativeness.
• Divergent panel selection and management strategies among agencies may impact survey estimates.

Purpose of the Study:

• To systematically analyze and compare differences in web survey results across multiple commercial survey agencies.
• To investigate the impact of varying panel management strategies on survey estimates for factual health items.

Main Methods:

• Six commercial survey agencies were commissioned to conduct the same web survey using an identical, standardized questionnaire.
• Surveys focused on factual health items and were fielded concurrently.
• A calibration approach was employed to adjust for demographic variations in the outcomes.

Main Results:

• Significant differences in health estimates were observed between probability and non-probability based web surveys.
• Post-survey weighting successfully reduced, but did not entirely eliminate, these estimate discrepancies.
• The variations observed between non-probability surveys, even after weighting, exceeded expectations for random samples from the same population.

Conclusions:

• Agency-specific panel strategies introduce measurable differences in web survey estimates.
• While weighting can mitigate some biases, it does not fully resolve discrepancies arising from non-probability sampling methods.
• Further research is needed to understand and standardize online panel management for more reliable survey data.