Automatic screening using word embeddings achieved high sensitivity and workload reduction for updating living...

Area of Science:

• Medical Informatics
• Biostatistics
• Evidence Synthesis

Background:

• Updating living network meta-analysis (NMA) requires screening numerous citations.
• Manual citation screening is time-consuming and labor-intensive.
• Efficient methods are needed to streamline the NMA update process.

Purpose of the Study:

• To develop and evaluate an algorithm for automatic citation screening during NMA updates.
• To assess the algorithm's performance in identifying eligible citations for full-text review.
• To quantify the reduction in workload and the rate of missed citations.

Main Methods:

• Developed a machine learning algorithm using word embeddings and logistic regression.
• Trained the algorithm on initially screened citations from four NMAs.
• Evaluated the algorithm on citations screened 2 years after the initial NMA conduct.
• Tested on NMAs from diverse medical domains.

Main Results:

• The algorithm achieved high sensitivity (94-100%) across four NMAs.
• It reduced manual screening workload by 53%, saving 1,345 citations from manual review.
• Only 2% of eligible citations were missed, with none impacting final NMA inclusion.

Conclusions:

• The developed algorithm effectively automates citation screening for NMA updates.
• It substantially decreases the workload associated with NMA maintenance.
• The algorithm offers a reliable method for efficient and accurate evidence synthesis updates.