Classification of Twitter Vaping Discourse Using BERTweet: Comparative Deep Learning Study

Area of Science:

• Computational linguistics
• Public health surveillance
• Machine learning applications

Background:

• Twitter data analysis is crucial for public health surveillance but manual categorization is labor-intensive.
• Existing machine and deep learning models require large annotated datasets, posing a barrier to research.
• Pretrained models like BERTweet offer higher quality with smaller annotated training sets.

Purpose of the Study:

• To develop and evaluate a BERTweet-based model for identifying vaping-related tweets, their commercial nature, and associated sentiment.
• To compare the performance of the BERTweet classifier against a Long Short-Term Memory (LSTM) model.

Main Methods:

• Collected and manually annotated 2401 English tweets related to vaping for relevance, commercial nature, and sentiment.
• Built three separate BERTweet classifiers using the annotated data.
• Trained and evaluated models using default parameters and a 10% hold-out set for testing.

Main Results:

• BERTweet classifiers achieved high performance: AUROC of 94.5% (relevance), 99.3% (commercial), and 81.7% (sentiment).
• Weighted F1 scores were 97.6% (relevance), 99.0% (commercial), and 86.1% (sentiment).
• BERTweet significantly outperformed the LSTM model across all classification categories.

Conclusions:

• Large, open-source deep learning models like BERTweet enable reliable and accurate classification of Twitter data for public health research.
• This approach enhances the utilization of Twitter data for faster exploration of time-sensitive information compared to traditional methods.
• BERTweet facilitates efficient identification of vaping-related content, commercial interests, and public sentiment on Twitter.