Using a Secure, Continually Updating, Web Source Processing Pipeline to Support the Real-Time Data Synthesis and...

Area of Science:

• Information Science
• Data Science
• Medical Informatics

Background:

• The COVID-19 pandemic created an "infodemic," overwhelming stakeholders with excessive data.
• Existing solutions struggle to synthesize heterogeneous web data in real-time for evidence-based decision-making.
• Clinicians and policymakers need efficient ways to process vast amounts of scientific literature.

Purpose of the Study:

• To develop a generic, real-time, continuously updating platform for scientific literature data synthesis and analysis.
• To validate the platform and curation methodology using COVID-19 medical literature.
• To expand the COVID-19 Open Research Dataset with new, unstructured data.

Main Methods:

• Developed a data pipeline using web crawler extraction and a novel human-in-the-loop curation methodology.
• The PanSurg Collaborative at Imperial College London implemented this approach.
• Characterized quality, relevance, and key evidence across diverse scientific literature sources.

Main Results:

• REDASA (Realtime Data Synthesis and Analysis) became a leading source of COVID-19 evidence, containing 104,000 documents.
• A foundational dataset of over 1400 curated articles was created in under two weeks.
• These articles represent approximately 10% of all COVID-19 papers, providing critical information.

Conclusions:

• REDASA's human-in-the-loop approach integrates a user-friendly platform with a natural language processing search engine.
• The platform provides a curated dataset in JSON format, aiding academic reviewers' critical appraisal.
• REDASA has captured one of the world's largest COVID-19 data corpora due to its extensive web crawling.