Multilingual Twitter Sentiment Classification: The Role of Human Annotators.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Automated sentiment classification of social media text, like Twitter, faces challenges in accuracy and generalizability.
• The impact of training data characteristics on model performance requires further investigation.

Purpose of the Study:

• To determine the limits of automated Twitter sentiment classification.
• To assess the influence of training data quality and size versus model architecture on classification performance.
• To establish methods for quantifying training data quality and its impact on model outcomes.

Main Methods:

• Analysis of a large, manually labeled multilingual Twitter dataset.
• Construction and comparison of various automated sentiment classification models.
• Application of annotator agreement measures to quantify training data quality.
• Evaluation of model performance against inter-annotator agreement benchmarks.

Main Results:

• Training data quality and size are more critical than the classification model type.
• No statistically significant performance difference was observed between top-performing models.
• Model performance converges with inter-annotator agreement as training data size increases.
• Regular monitoring of self- and inter-annotator agreements enhances training datasets and model performance.

Conclusions:

• The effectiveness of automated sentiment analysis on Twitter is primarily dictated by the training data.
• Continuous quality assessment of annotation is essential for improving sentiment classification models.
• Human perception of sentiment classes (negative, neutral, positive) is ordered, a factor to consider in model design.