Development of a global infectious disease activity database using natural language processing, machine learning, and...

Area of Science:

• Computational epidemiology
• Public health informatics
• Machine learning applications

Background:

• Traditional infectious disease surveillance relies on established public health reporting systems.
• These systems can have delays in identifying and reporting disease outbreaks.
• The increasing volume of online media presents a potential source for early outbreak detection.

Purpose of the Study:

• To evaluate the efficacy of machine learning (ML) for extracting infectious disease activity from online media reports.
• To develop and validate a real-time disease activity surveillance system using ML.

Main Methods:

• A dataset of 8,322 labeled media reports was curated to train a classifier.
• A held-out test set was used for validation.
• The system's performance was compared against World Health Organization (WHO) Disease Outbreak News reports.

Main Results:

• The ML classifier achieved a recall of 88.8% and precision of 86.1%.
• The surveillance system detected 94% of WHO-identified outbreaks reported in online media.
• Detection occurred an average of 43.4 days earlier than traditional reporting.

Conclusions:

• Machine learning, combined with natural language processing and human expertise, enables efficient identification of disease activity from digital media.
• A global, real-time disease activity database has been established, surveilling 114 illnesses and syndromes.
• Further assessment of the system's bias, representativeness, granularity, and accuracy is required.