Ensuring the quality and specificity of preregistrations

Area of Science:

• Psychology
• Research Methodology
• Open Science

Background:

• Researchers make numerous choices during the research process, termed 'researcher degrees of freedom'.
• Opportunistic use of these degrees of freedom can lead to false-positive results and inflated effect sizes.
• Preregistration aims to mitigate these issues by pre-specifying research plans.

Purpose of the Study:

• To compare the effectiveness of two preregistration formats on the Open Science Framework (OSF): Standard (unstructured) and Prereg Challenge (structured).
• To assess whether structured preregistration better restricts researcher degrees of freedom compared to unstructured formats.

Main Methods:

• A comparative study analyzing 53 preregistrations from each of the two OSF formats (Standard and Prereg Challenge).
• Evaluation focused on the degree to which each format restricted the opportunistic use of researcher degrees of freedom.
• Assessed inter-coder concordance on hypothesis identification.

Main Results:

• The structured Prereg Challenge format restricted researcher degrees of freedom more effectively than the unstructured Standard format (Cliff's Delta = 0.49).
• Neither format completely eliminated researcher degrees of freedom.
• Low inter-coder agreement (14%) on the number of hypotheses indicated poor clarity in stated hypotheses.

Conclusions:

• Effective preregistration is challenging, requiring careful consideration of format and guidance.
• Structured preregistration formats with clear instructions and review appear more effective in limiting researcher degrees of freedom.
• Improving the clarity of hypothesis statements is crucial for effective preregistration and research integrity.