An unsupervised machine learning model for discovering latent infectious diseases using social media data

Area of Science:

• Computational epidemiology
• Machine learning applications in public health
• Social media analytics for disease surveillance

Background:

• Traditional infectious disease surveillance relies on formalized data, which is time-consuming for emerging or latent diseases.
• Existing social media analysis methods often require prior knowledge of disease names and symptoms (top-down approach).
• Latent infectious diseases, not yet recognized by health institutes, pose a challenge for timely public health intervention.

Purpose of the Study:

• To propose an unsupervised machine learning model for identifying latent infectious diseases from social media data.
• To develop a bottom-up approach for disease discovery without requiring pre-existing information on disease names or symptoms.
• To enable faster detection of potential outbreaks compared to formalization by public health institutes.

Main Methods:

• Extraction of social media messages with user and temporal information.
• Application of unsupervised sentiment analysis to identify user expressions of symptoms.
• Creation of symptom weighting vectors based on sentiment and social media expressions for individuals and time periods.
• Retrieval of latent infectious disease information from aggregated symptom data.

Main Results:

• Validation using Twitter data (August 2012 - May 2013) against electronic medical records for influenza.
• Achieved high performance metrics: precision (0.773), recall (0.680), and F1-score (0.724).
• Demonstrated the model's capability to identify disease-related information from unlabeled social media data.

Conclusions:

• The unsupervised model effectively identifies latent infectious diseases using social media data without prior information.
• This approach offers a faster alternative to traditional disease formalization by public health bodies.
• The model leverages user, textual, and temporal social media information, alongside sentiment analysis, for localized disease detection.