Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the...

Area of Science:

• Bibliometrics and Information Science
• Machine Learning Applications in Health Research
• Evidence Synthesis and Systematic Reviews

Background:

• Maintaining the Cochrane COVID-19 Study Register (CCSR) involves a substantial workload for identifying relevant research studies.
• Machine learning (ML) offers a potential solution to automate and optimize the study identification process.

Purpose of the Study:

• To develop, calibrate, and evaluate a machine learning classifier to reduce the manual screening workload for the CCSR.
• To improve the efficiency of identifying COVID-19 research studies for inclusion in the register.

Main Methods:

• Developed a ML classifier ('Cochrane COVID-19 Study Classifier') using manually labeled title-abstract records.
• Calibrated the classifier to achieve a recall of at least 99% using a dedicated dataset.
• Evaluated the classifier's performance on a separate dataset to assess its accuracy and impact on workload reduction.

Main Results:

• The classifier was trained on 59,513 records and calibrated using 16,123 records.
• Achieved a recall of 98.9% (missing only 1% of eligible records) with a precision of 0.638.
• Resulted in a net screening workload reduction of 24.1%, excluding 1113 records correctly.

Conclusions:

• The Cochrane COVID-19 Study Classifier effectively reduces manual screening workload for identifying COVID-19 research studies.
• The classifier demonstrates a very low and acceptable risk of missing eligible studies.
• The tool is now operational in the live workflow for the Cochrane COVID-19 Study Register.