A biomedically oriented automatically annotated Twitter COVID-19 dataset.

Area of Science:

• Biomedical Informatics
• Computational Social Science
• Public Health Informatics

Background:

• Social media, particularly Twitter, is increasingly used for biomedical research.
• The COVID-19 pandemic highlighted the need for real-time clinical data from non-traditional sources.
• Manual annotation of social media data is costly, time-consuming, and often results in small, non-generalizable datasets.

Purpose of the Study:

• To develop and release a large-scale, automatically annotated dataset of tweets for biomedical research.
• To address the limitations of manually curated datasets in terms of cost, size, and generalizability.
• To facilitate near-real-time analysis of diseases, interventions, and sequelae using social media data.

Main Methods:

• Leveraged best-practices for identifying tweets with potential clinical relevance.
• Evaluated multiple SpaCy-based annotation frameworks against a manually annotated gold-standard dataset.
• Selected the optimal automatic annotation method and applied it to over 120 million tweets.

Main Results:

• A publicly released dataset of 120 million automatically annotated tweets for biomedical research.
• Demonstrated the feasibility of large-scale automatic annotation for clinical relevance.
• Established a benchmark for automatic tweet annotation in the biomedical domain.

Conclusions:

• The developed dataset provides a valuable, scalable resource for biomedical research.
• Automatic annotation offers a cost-effective solution to the data scarcity problem in social media research.
• This work enables broader and more efficient use of social media data for public health surveillance and clinical studies.