How to evaluate sentiment classifiers for Twitter time-ordered data?

Area of Science:

• Natural Language Processing
• Machine Learning
• Computational Social Science

Background:

• Social media platforms like Twitter are vital for gauging public sentiment on diverse topics.
• Evaluating sentiment classification models for time-ordered data presents unique challenges due to temporal dependencies.

Purpose of the Study:

• To investigate reliable performance estimation procedures for sentiment classifiers using Twitter data.
• To compare the effectiveness of cross-validation and sequential validation methods in time-ordered scenarios.

Main Methods:

• Collected 1.5 million tweets across 13 European languages.
• Developed 138 sentiment models and associated datasets.
• Empirically compared six estimation procedures: three cross-validation variants and three sequential validation variants.

Main Results:

• No significant performance difference was found between the best cross-validation and sequential validation methods.
• Cross-validation variants tended to overestimate classifier performance.
• Sequential validation methods tended to underestimate performance, offering a more conservative estimate.

Conclusions:

• Standard cross-validation with random data splitting is unsuitable for time-ordered data.
• Blocked cross-validation and sequential validation are more appropriate for evaluating sentiment classifiers on temporal data streams.
• Sequential validation provides a more realistic, albeit potentially underestimated, view of real-world performance.