Reproducibility of Published Randomized Controlled Trial Results in Cardiovascular Surgery: Protocol for a...

Reproducible study findings are those that are shown to be identical or nearly identical when an original study is repeated using similar methods. Findings from many randomized controlled trials (RCTs) in cardiovascular surgery have not been reproduced or are small, statistically fragile, and potentially influenced by funding source. They also demonstrate variable end point selection and protocol adherence, all of which may influence reproducibility of their reported results.

This study aims to assess the reproducibility of RCTs conducted in the most commonly studied disease states in cardiovascular surgery that compared at least one open surgical procedure to an alternative open, interventional catheter-based, or sham procedure or medical therapy.

We will conduct a meta-epidemiological study of the reproducibility of RCTs using umbrella review methodology. We designed the study protocol using the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) statement and published recommendations on conducting umbrella reviews. We will search MEDLINE, Embase, and the CENTRAL from their inception for systematic reviews published in the English language that included data from phase 3 RCTs or those with a clinically important primary outcome and compared an open cardiovascular surgery (performed in the most commonly studied disease states across these specialties) to either an alternative open, catheter-based, or sham procedure or medical therapy. Two investigators will independently review the individual RCTs from these systematic reviews and extract data and assess risk of bias (using the Cochrane Risk of Bias-2 tool), the fragility or reverse fragility indices (which provide a measurement of the statistical robustness of the trials), and the authors' interpretation of the results (also referred to as spin). RCTs will be classified as the original study if it was the first chronological study (based on participant enrollment date) to evaluate the compared interventions and as reproduction attempts if they reevaluate the compared interventions in a different set of participants. The interpretation of a study's results will be based on the findings of its primary end point and be used to assess reproducibility. We will also create multivariable regression models to determine whether the adjusted odds of reproducibility are independently predicted by 12 different variables.

We registered the study protocol but have not yet conducted the search. We aim to complete the study by the end of 2026 and submit for peer review by the summer of 2027.

This study protocol may be used as a framework for future reproducibility research. The results will help guide which cardiovascular surgery practices warrant careful interpretation and which are in need of reproduction studies. They will also identify characteristics of an original RCT in this field that predicts future study reproducibility before reproducibility studies are published.