Curating Cyberbullying Datasets: a Human-AI Collaborative Approach.

Area of Science:

• Computer Science
• Social Science
• Psychology

Background:

• Cyberbullying detection models require high-quality, unbiased datasets.
• Human annotation of cyberbullying data faces challenges like annotator bias and subjectivity.
• Existing datasets may be inadequate for training reliable machine learning models.

Purpose of the Study:

• To propose and evaluate machine learning methods for filtering ambiguous comments in cyberbullying datasets.
• To curate new, unambiguous datasets for improved cyberbullying detection.
• To enhance the performance of artificial neural networks in identifying online aggression.

Main Methods:

• Developed two machine learning approaches to identify and filter unambiguous comments.
• Utilized consensus filtering based on human annotator majority and algorithmic agreement.
• Curated new datasets from a YouTube cyberbullying corpus annotated via Amazon Mechanical Turk (AMT).

Main Results:

• The proposed filtering method successfully identified unambiguous comments.
• Classifiers trained on curated datasets showed significant performance improvements.
• The approach provided insights into consistently classified bullying and non-bullying content.

Conclusions:

• Machine learning-driven data curation enhances cyberbullying detection model performance.
• Consensus filtering effectively addresses issues of bias and subjectivity in dataset annotation.
• This annotation strategy is adaptable to other classification tasks with complex datasets.