Representativeness and response validity across nine opt-in online samples

The study of human behaviour has shifted in the past 15 years. It increasingly relies on opt-in non-probability online data sources. Here we offer an analysis of nine such sources (N = 13,053), aiming to inform researchers conducting experiments or correlational studies. We assess response validity (attentiveness, effort, honesty, speeding and attrition), the extent to which samples represent the underlying population (demographics, attitude representativeness and experimental treatment effects) and professionalism (number and frequency of studies taken and modality of device on which studies are taken). We document substantial variation across samples on each dimension. Samples using demographic quotas display relatively higher amounts of representativeness across multiple indicators (beyond demographics) but often exhibit less response validity. However, inclusion of two attention checks early in a survey can enhance response validity without negatively impacting representativeness. We offer guidance for choosing opt-in samples, depending on the purpose of the research and resource constraints.