Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling...

Area of Science:

• Bibliometrics
• Information Science
• Computational Biology

Background:

• Systematic reviews are crucial for synthesizing research but manual title and abstract screening is labor-intensive.
• Existing automation attempts face limitations like domain-specificity and reliance on labeled data.
• Developing statistical methods for automated screening is essential to improve efficiency.

Purpose of the Study:

• To develop statistical methodology for automated title and abstract screening in systematic reviews.
• To retrospectively apply and evaluate the methodology on existing systematic review datasets.
• To characterize the performance of the automated screening algorithm through simulation.

Main Methods:

• Implemented a Latent Dirichlet Allocation (LDA)-based topic model for deriving document topics from titles and abstracts.
• Defined a score threshold for classifying documents as relevant or not relevant for full-text review.
• Utilized search keywords, including database retrieval terms, to derive document scores.

Main Results:

• In a helminth dataset case study, sensitivity of [Formula: see text] and a false positive rate of [Formula: see text] were achieved.
• In a Wilson disease dataset case study, sensitivity of [Formula: see text] and specificity of [Formula: see text] were achieved.
• The methodology demonstrated a specificity of approximately [Formula: see text] on tested data.

Conclusions:

• Unsupervised title and abstract screening offers potential for reducing systematic review workload.
• The developed methodology achieved high specificity but low sensitivity, necessitating user awareness of potential limitations.
• Incorporating additional targeted search keywords and using automated screening as a supplementary tool to manual screening are recommended strategies.