Comparing the performance of narrow vs. broad search strategies when using machine learning-based software for...

Area of Science:

• Information Science
• Health Sciences
• Systematic Reviews

Background:

• Systematic reviews increasingly utilize semi-automated screening tools.
• Optimizing database search strategies is crucial for efficient systematic review screening.
• Active learning tools aim to reduce screening workload and improve efficiency.

Purpose of the Study:

• To evaluate the workload and recall performance of narrower versus broader database search strategies.
• To assess the impact of different search strategies within active learning screening tools.
• To explore the effect of stopping rules on screening workload and recall.

Main Methods:

• Retrospective analysis of 10 completed systematic reviews.
• Utilized ASReview LAB software for simulating screening processes.
• Compared broader (more sensitive) and narrower (more precise) search strategies.
• Recorded metrics like records screened to achieve 95% recall.
• Investigated the influence of three stopping rules.

Main Results:

• For quantitative reviews, the difference in screening time between broader and narrower strategies was minimal (≤35 minutes).
• Broader strategies increased workload for qualitative and other review types.
• Time-based stopping rules significantly increased workload with broader search strategies.

Conclusions:

• Time savings from semi-automated tools may be offset by increased screening time with broader, more sensitive search strategies.
• Librarians and information specialists must balance search sensitivity and specificity when using these tools.
• Careful consideration of search strategy parameters is essential for efficient systematic review screening.