A Demonstration of Propensity-Score Weighting to Adjust a Social Media Nonprobability Sample Survey of Political...

Interest in using nonprobability online samples continues to grow despite concerns about selection bias. Many methods exist for adjusting nonprobability data so it may yield generalizable inferences. Here we investigate whether a propensity score weighting method can balance differences between a probability sample and a nonprobability sample of Twitter (now X) users to evaluate the feasibility of using social media data for producing generalizable inferences on public opinion. We fielded identical surveys to 2,001 probability-sampled respondents (June 30-July 22, 2022) and 949 Twitter users (March 1-July 13, 2022); final analytic sample sizes were 1,972 and 822, respectively. The nonprobability sample differed significantly in demographic characteristics (younger, lower income, higher educational attainment), and broadly endorsed significantly more liberal attitudes toward a range of political and policy issues than the probability sample. We show that the propensity score weighting procedure, using demographics, techno/psychographics, and political ideology, reconciles differences between the samples for 25 of the 27 attitudes assessed. The results demonstrate the feasibility and utility of the propensity score weighting procedure to replicate a probability sample with nonprobability social media data and add to the literature on the use of nonprobability samples to draw population-level inferences.