Creating a Bot-tleneck for malicious AI: Psychological methods for bot detection

Area of Science:

• Computer Science
• Artificial Intelligence
• Psychology

Background:

• Traditional bot detection methods like CAPTCHAs are increasingly ineffective against advanced AI.
• Manual analysis of bot behavior is labor-intensive and inefficient.
• Generative AI poses a future threat to current bot-screening techniques.

Purpose of the Study:

• To develop and evaluate automated bot-screening questions grounded in psychological research.
• To create a proactive screen against sophisticated bots.
• To assess the efficacy of novel bot-screeners against existing methods.

Main Methods:

• Developed automated, psychologically-grounded bot-screening questions.
• Recruited MTurkers for a Qualtrics survey.
• Compared bot identification rates between novel screeners, manual analysis, and Google's reCAPTCHA V3.

Main Results:

• Novel bot-screeners identified 18.9% of participants as potential bots, significantly higher than reCAPTCHA V3's 1.7%.
• The developed automated questions demonstrated superior performance compared to CAPTCHAs.
• Analysis revealed varying strengths and weaknesses among the novel bot-screener types.

Conclusions:

• Automated, psychologically-grounded questions offer a more effective approach to bot detection than CAPTCHAs.
• Current bot-screening methods require significant improvement to counter advanced AI.
• The developed bot-screeners show promise for proactive and efficient bot mitigation.