The False positive problem of automatic bot detection in social science research

Area of Science:

• Computational Social Science
• Network Science
• Machine Learning Evaluation

Background:

• Botometer is a widely used tool for estimating bot prevalence in online discourse.
• Accurate bot identification is crucial for understanding social media platforms and academic research.
• Previous studies have relied on Botometer without fully assessing its long-term diagnostic accuracy.

Purpose of the Study:

• To evaluate the diagnostic accuracy and temporal stability of Botometer scores.
• To investigate Botometer's performance across different languages (English/German).
• To highlight the implications of Botometer's limitations for social science research.

Main Methods:

• Collected Botometer scores for 4,134 accounts (3,000 bots, 1,134 humans) over three months.
• Analyzed scores in both English and German datasets.
• Examined the variance in Botometer scores and classification thresholds over time.

Main Results:

• Botometer scores demonstrated imprecision in bot estimation, particularly for non-English data.
• Temporal analysis revealed significant variance in Botometer thresholds, leading to false positives and negatives.
• A notable number of human users were misclassified as bots, and vice versa.

Conclusions:

• Botometer's accuracy is questionable, especially across different languages and over time.
• The unreliability of Botometer has significant consequences for social science research relying on its classifications.
• Computational social scientists must critically evaluate machine learning tools for bot identification.