Automated citation searching in systematic review production: A simulation study

Area of Science:

• Information Science
• Bibliometrics
• Systematic Review Methodology

Background:

• Bibliographic aggregators like OpenAlex and Semantic Scholar offer potential for automating citation searching in systematic reviews.
• Automated methods may increase efficiency in systematic review production.

Purpose of the Study:

• To evaluate the performance of automated citation searching against standard strategies.
• To identify factors influencing the performance of automated citation searching.

Main Methods:

• Automated citation searching was simulated on 27 systematic reviews across OpenAlex and Semantic Scholar.
• Performance metrics included recall, precision, and F1-F3 scores, compared to original review search strategies.
• Factors analyzed included study area, number of articles, seed article characteristics, and API choice.

Main Results:

• Automated citation searching outperformed standard strategies in precision (p < 0.05) and F1 score (p < 0.05).
• Recall (p < 0.05) and F3 score (p < 0.05) were not superior to standard strategies.
• Performance varied by study area, with higher performance in environmental management than social policy.

Conclusions:

• Automated citation searching is best used as a supplementary strategy when precision is prioritized over recall.
• Its superior precision and F1 score suggest utility in specific systematic review contexts.
• Lower recall and F3 scores indicate limitations when comprehensive retrieval is paramount.