COVID-Scraper: An Open-Source Toolset for Automatically Scraping and Processing Global Multi-Scale Spatiotemporal...

Area of Science:

• Epidemiology
• Data Science
• Public Health

Background:

• The COVID-19 pandemic necessitated extensive research into virus spread and societal impact.
• Existing COVID-19 data sources lack standardization, hindering international fine-scale research.
• Challenges include varied data formats, update frequencies, and limited access to original datasets.

Purpose of the Study:

• To develop an automated toolset for extracting, refining, and unifying global COVID-19 case data.
• To provide a standardized, high-quality, and accessible international dataset for COVID-19 research.
• To enable real-time dynamic data access with a global perspective.

Main Methods:

• Cloud-based web scraping techniques were employed for data extraction.
• Data refinement and unification processes were implemented to ensure consistency.
• The toolset automatically processes data from multiple countries at various scales.

Main Results:

• A comprehensive, standardized, and updated international COVID-19 dataset was created.
• The toolset enables public access to real-time dynamic COVID-19 data.
• Two case studies demonstrate the utility of the generated datasets.

Conclusions:

• The developed toolset effectively addresses the challenge of fragmented international COVID-19 data.
• The open-source nature allows for future extensions and broader applications.
• Provides a valuable resource for epidemiological modeling, prevention strategy development, and impact analysis.