Reporting software tools and automation in the protocols of evidence syntheses: meta-research study

The growing volume of primary research and the increasing demand for timely, high-quality evidence syntheses (ESs) have intensified interest in using software, automation, and artificial intelligence. While guidance such as PRISMA 2020 encourages reporting automation in completed ESs, there is little emphasis on documenting planned automation at the protocol stage. This gap is critical, as protocols are intended to ensure transparency, reduce research waste, and safeguard methodological rigor. The objective of this metaresearch study was to investigate how software tools and automation are reported in published ES protocols, how their planned use is distributed across ES phases and subphases, and how these plans align with the reporting of subsequently published ESs.

We conducted a cross-sectional analysis of ES protocols published in August 2023 and indexed in MEDLINE. Protocols of any ES type were eligible. Two reviewers independently screened studies and extracted data on reporting standards, software tools, and mentions of automation across predefined ES phases and subphases. Descriptive statistics were used for analysis. For additional insight, we identified ESs published by January 2026 that corresponded to included protocols and assessed concordance between them.

Sixty-eight protocols describing 73 planned ESs were included. Nearly all protocols (95.6%) planned to use at least 1 software tool, with a mean of 2.44 tools per protocol; however, only 8.8% explicitly reported automation for specific subphases. Planned tool use was concentrated in database searching and record download, while no protocol planned automation across all ES phases. Reporting standards were frequently cited, yet adherence and checklists use were inconsistently documented. Among the 33 ESs subsequently published, deviations from protocols were common and not often reported.

Although most ES protocols anticipated using software tools, explicit and transparent reporting of automation remained rare and incomplete. Discrepancies between planned and actual tool use in completed ESs further undermine research integrity. These findings highlight the need for clearer guidance and reporting requirements for software and automation at the protocol stage. Extending reporting standards to include automation-specific items could strengthen transparency, improve methodological rigor, and enhance the responsible integration of automation in ES.