Geolocation to Identify Online Study-Eligible Gay, Bisexual, and Men who have Sex with Men in Philadelphia,...

Data collection and cleaning procedures to exclude bot-generated responses are used to maintain the data integrity of samples from online surveys. However, these procedures may be time-consuming and difficult to implement. Thus, we aim to evaluate the validity of a single-step geolocation algorithm for recruiting eligible gay, bisexual, and men who have sex with men in Philadelphia for an online study.

We used a 4-step approach, based on common practices for evaluating bot-generated and fraudulent responses, to assess the validity of participants' Qualtrics survey data as our referent standard. We then compared it to Qualtrics' single-step geolocation algorithm that used the MaxMind commercial database to map participants' Internet protocol address to their approximate location. We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the single-step geolocation approach relative to the 4-step approach.

There were 826 respondents who completed the survey and 440 (53%) were eligible for enrollment based on the 4-step approach. The single-step geolocation approach yielded a sensitivity of 91% (95% CI = 88%, 93%), specificity of 79% (95% CI = 74%, 83%), PPV of 83% (95% CI = 80%, 86%), and NPV of 88% (95% CI = 85%, 91%).

Geolocation alone provided a moderately high level of agreement with the 4-step approach for identifying geographically eligible participants in the online sample, but both approaches may be subject to additional misclassification. Researchers may want to consider multiple procedures to ensure data integrity in online samples.