SEED: Symptom Extraction from English Social Media Posts using Deep Learning and Transfer Learning

Area of Science:

• Digital epidemiology
• Natural Language Processing (NLP)
• Public Health Informatics

Background:

• Social media platforms generate vast amounts of colloquial health-related data, posing challenges for traditional analysis.
• Extracting accurate symptom and disease mentions from informal text is crucial for population health monitoring.
• Existing methods struggle with the nuances and scarcity of specific health information in social media.

Approach:

• Developed SEED, a novel NLP approach for detecting and normalizing symptom and disease mentions.
• Utilized multi-corpus training and deep learning models for robust performance.
• Tested SEED on social media data from Twitter and DailyStrength, normalizing to Unified Medical Language System (UMLS) terminology.

Key Points:

• SEED achieved high F1 scores (0.86 on DailyStrength, 0.72 on Twitter), outperforming previous methods.
• The system demonstrated effectiveness in identifying COVID-19 symptoms, including those not present in training data.
• Multi-corpus training significantly enhances performance and adaptability.

Conclusions:

• SEED offers a powerful solution for mining colloquial health data from social media for digital epidemiology.
• Continuous training is essential to maintain performance due to evolving social media language.
• The approach holds significant potential for real-time disease surveillance and public health insights.