Using an Open Science Checklist in Grant Proposal Reviews to Predict Reproducibility of Funded Publications.

Area of Science:

• Scientific publishing
• Research integrity
• Open science practices

Background:

• Reproducibility is crucial for scientific advancement.
• Assessing reproducibility early in the research process, such as during grant proposal review, is challenging.
• The effectiveness of tools like checklists in predicting reproducibility is not well-established.

Purpose of the Study:

• To evaluate the utility of an Open Science checklist for grant referees in predicting the reproducibility of resulting publications.
• To compare the predictive accuracy of referees using the checklist versus those not using it.

Main Methods:

• Comparative accuracy study using funded grant proposals.
• Two groups of mock referees assessed proposals: one with an Open Science checklist, one without.
• Actual reproducibility of resulting publications was attempted.
• Sensitivity, specificity, and predictive values were calculated.

Main Results:

• Only 6.9% of publications were reproducible.
• The checklist group predicted 16.8% reproducibility (PPV 11.8%), while the non-checklist group predicted 75.2% (PPV 7.9%).
• Neither group accurately predicted which proposals would lead to reproducible findings.

Conclusions:

• The Open Science checklist demonstrated low positive predictive value for predicting reproducibility.
• The study suggests that current methods, including the checklist, are insufficient for reliably predicting research reproducibility at the grant proposal stage.
• Further research is needed to improve the prediction of research reproducibility.