A longitudinal study of topic classification on Twitter.

Area of Science:

• Computational Social Science
• Natural Language Processing
• Machine Learning

Background:

• Twitter is a vast source of information on diverse topics.
• Classification-based filtering can personalize content for users.
• The long-term efficacy and robustness of these filters are not well understood.

Purpose of the Study:

• To evaluate the long-term generalization and robustness of topic classifiers trained on Twitter data.
• To identify critical features for sustained classifier performance over time.
• To assess the stability of classifier performance across various topics and time horizons.

Main Methods:

• Collected over 800 million English Tweets via the Twitter streaming API (2013-2014).
• Trained topic classifiers for 10 diverse themes.
• Analyzed classifier performance over extended periods, focusing on generalization to novel content and feature importance.

Main Results:

• Classifiers generalize to novel topical content with high precision over a year, but performance degrades over time.
• Hashtags and simple terms are the most informative feature classes.
• Removing training hashtags from validation sets improves generalization.
• Tweet volume per user is a stronger indicator of informativeness than follower/friend count.

Conclusions:

• Classification-based topical filtering on Twitter is a justified approach.
• Understanding feature properties is critical for optimizing long-term classifier performance.
• Classifier robustness and generalization capabilities are demonstrated, with identified areas for improvement.