The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and...

Area of Science:

• Information Science
• Medical Informatics
• Bibliometrics

Background:

• Active learning is increasingly used for screening large datasets in systematic reviews and meta-analyses.
• The goal of active learning is to identify relevant records efficiently by improving predictions on unlabeled data.
• A key challenge is determining the optimal stopping point to balance labeling costs and the risk of excluding relevant records.

Purpose of the Study:

• To introduce the SAFE procedure, a set of practical stopping heuristics for active learning in systematic review screening.
• To provide clear guidelines for deciding when to end the active learning process in screening software.
• To minimize the risk of missing relevant papers during the screening process.

Main Methods:

• The study proposes the SAFE (Stopping Appropriately for Efficiency) procedure.
• This procedure utilizes a mix of stopping heuristics to guide the termination of active learning.
• It aims to balance the cost of continued screening with the risk of erroneous exclusions.

Main Results:

• The SAFE procedure offers a practical and conservative approach to ending active learning.
• It helps reviewers make informed decisions by balancing screening costs against the risk of missing relevant records.
• The heuristics are designed to minimize the chance of excluding valuable papers.

Conclusions:

• The SAFE procedure provides a valuable tool for optimizing active learning in systematic reviews.
• It addresses the critical challenge of determining when to stop the screening process effectively.
• The decision to stop active learning requires careful consideration of dataset specifics and potential model errors.