A new method for testing reproducibility in systematic reviews was developed, but needs more testing.

Area of Science:

• Medical Research Methodology
• Evidence Synthesis

Background:

• Systematic reviews (SRs) are crucial for evidence-based practice.
• Ensuring the reproducibility of SRs is essential for their reliability and validity.
• Current methods for assessing SR reproducibility are limited.

Purpose of the Study:

• To develop and evaluate a retrospective approach for testing the reproducibility of systematic reviews.
• To assess the feasibility of replicating key components of SRs, including literature searches, study selection, data extraction, and risk of bias assessments.

Main Methods:

• A case study approach was employed to test the developed reproducibility assessment method.
• Replication of literature searches, a 25% random sample for study selection, data extraction, and risk of bias (ROB) assessments by two independent reviewers.
• Narrative comparison of replicated results with the original systematic review.

Main Results:

• Full reproduction of the original literature search was not achieved, resulting in minor discrepancies in retrieved citations.
• The most significant disagreements between reviewers occurred during the study selection phase.
• Risk of bias assessment presented the greatest challenge due to insufficient reporting of clear criteria, although satisfactory agreement was reached.

Conclusions:

• The developed approach provides a framework for assessing systematic review reproducibility.
• Further testing and comparison with other methods are necessary as the field of SR reproducibility testing is still developing.
• Improvements in reporting standards for risk of bias assessments are needed to enhance reproducibility.