Unsupervised [randomly responding] survey bot detection: In search of high classification accuracy

Area of Science:

• Social Sciences
• Psychometrics
• Data Science

Background:

• Online surveys are widely used in social sciences.
• Bot prevalence threatens data quality and research integrity.
• Detecting bots is crucial when preventative measures fail.

Purpose of the Study:

• To evaluate a new unsupervised algorithm for detecting bots in survey data.
• To understand factors influencing the algorithm's classification accuracy.
• To identify conditions for reliable bot detection.

Main Methods:

• Utilized simulations with hypothetical human responses from item response theory models.
• Contaminated real human data with simulated bots across 35 datasets.
• Employed a permutation test assuming item exchangeability for bots but not humans.

Main Results:

• Algorithm accuracy is sensitive to item properties, number of items, latent factors, and factor correlations.
• High classification accuracy (around 95%+) achieved under optimal conditions.
• Lower accuracy observed under certain conditions, highlighting performance variability.

Conclusions:

• The model-agnostic, unsupervised algorithm shows promise for bot detection.
• Classification accuracy is enhanced by more items, more response categories, and item difficulty variation.
• Researchers must consider these factors to ensure reliable bot detection and maintain data integrity.