Model-agnostic unsupervised detection of bots in a Likert-type questionnaire

Area of Science:

• Computational Social Science
• Survey Methodology
• Machine Learning

Background:

• Existing bot detection methods for online surveys often rely on labeled data or specific measurement models.
• A gap exists for bot detection when neither labeled data nor a predefined measurement model is available.
• This is particularly relevant for inventories with uniform Likert-type response scales across all items.

Purpose of the Study:

• To propose a novel bot detection algorithm applicable in unsupervised and model-agnostic settings.
• To address the challenge of identifying bots in online survey data lacking labeled responses or measurement models.
• To provide a robust method for inventories with consistent Likert-type category numbers.

Main Methods:

• Developed a bot detection algorithm based on permutation tests.
• Incorporated leave-one-out calculations of outlier statistics for each respondent.
• The algorithm generates a p-value to test the null hypothesis that a respondent is a bot.

Main Results:

• The proposed algorithm demonstrated robust nominal sensitivity calibration, independent of bot response distributions.
• Simulation studies showed significant improvements over naive alternatives in 95% sensitivity calibration.
• The algorithm also enhanced classification accuracy in numerous simulated scenarios.

Conclusions:

• The novel permutation test-based algorithm effectively detects bots in online surveys without requiring labeled data or strong model assumptions.
• This unsupervised, model-agnostic approach offers improved sensitivity calibration and classification accuracy.
• The method is particularly valuable for survey data with uniform Likert-type scales.