Related Experiment Video
Updated: Jul 29, 2025

Collecting Sleep, Circadian, Fatigue, and Performance Data in Complex Operational Environments
Published on: August 8, 2019
Scientists' perception of pilot study quality was influenced by statistical significance and study design
Lauren von Klinggraeff1, Sarah Burkart1, Christopher D Pfledderer1
1Department of Exercise Science, University of South Carolina, Columbia, SC, USA.
Objectives:
Preliminary studies play a key role in developing large-scale interventions but may be held to higher or lower scientific standards during the peer review process because of their preliminary study status.
Study Design And Setting:
Abstracts from 5 published obesity prevention preliminary studies were systematically modified to generate 16 variations of each abstract. Variations differed by 4 factors: sample size (n = 20 vs. n = 150), statistical significance (P < 0.05 vs. P > 0.05), study design (single group vs. randomized 2 groups), and preliminary study status (presence/absence of pilot language). Using an online survey, behavioral scientists were provided with a randomly selected variation of each of the 5 abstracts and blinded to the existence of other variations. Respondents rated each abstract on aspects of study quality.
Results:
Behavioral scientists (n = 271, 79.7% female, median age 34 years) completed 1,355 abstract ratings. Preliminary study status was not associated with perceived study quality. Statistically significant effects were rated as more scientifically significant, rigorous, innovative, clearly written, warranted further testing, and had more meaningful results. Randomized designs were rated as more rigorous, innovative, and meaningful.
Conclusion:
Findings suggest reviewers place a greater value on statistically significant findings and randomized control design and may overlook other important study characteristics.
More Related Videos
06:26Meta-analysis of Voxel-Based Neuroimaging Studies using Seed-based d Mapping with Permutation of Subject Images SDM-PSI
Published on: November 27, 2019
08:33A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
Related Concept Videos
Statistical Significance
Study Design in Statistics
Does aspirin reduce the risk of heart attacks? Is one brand of fertilizer more effective at growing roses than another? Is fatigue as dangerous to a driver as the influence of alcohol? Questions like these are answered using randomized experiments with proper...
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Significance Testing: Overview
Bias in Epidemiological Studies
Accuracy and Errors in Hypothesis Testing
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...