Related Experiment Video
Updated: Aug 8, 2025
06:37
Author Spotlight: Addressing Technical and Subjective Challenges in Measuring Classroom Attention
Published on: December 15, 2023
3.9K
Supervised Classes, Unsupervised Mixing Proportions: Detection of Bots in a Likert-Type Questionnaire.
Michael John Ilagan1, Carl F Falk1
1McGill University, Montreal, Quebec, Canada.
Educational and Psychological Measurement
|March 3, 2023
Summary
Detecting bots in online surveys is challenging. The new SCUMP algorithm improves bot detection accuracy by estimating contamination rates, ensuring reliable data even with high bot presence.
Area of Science:
- Psychometrics
- Data Science
- Online Research Methods
Background:
- Online surveys risk data contamination from bots.
- Existing bot detection methods (NRIs) lack universal cutoffs.
- High-specificity cutoffs are inaccurate with high contamination rates.
Purpose of the Study:
- To propose a novel algorithm, SCUMP, for accurate bot detection in online surveys.
- To develop a method for selecting optimal cutoffs that maximize accuracy.
- To address the limitations of existing bot detection techniques.
Main Methods:
- Developed the Supervised Classes, Unsupervised Mixing Proportions (SCUMP) algorithm.
- Utilized a Gaussian mixture model for unsupervised estimation of contamination rates.
- Conducted a simulation study to evaluate algorithm performance across varying contamination levels.
Main Results:
- SCUMP algorithm effectively estimates sample contamination rates.
- Proposed cutoffs maintained high accuracy across diverse contamination rates.
- The method demonstrated robustness in simulation studies.
Conclusions:
- SCUMP offers an accurate and adaptable approach to bot detection in online surveys.
- This algorithm enhances data integrity by improving the identification of malicious responses.
- SCUMP provides a valuable tool for researchers dealing with potentially contaminated online data.
