Related Experiment Video
Updated: Aug 16, 2026
The Joint Effect of Social Comparison and Social Distance on Evaluation of Intertemporal Choice Outcomes in Event-related Potential Studies
Published on: August 25, 2023
The undue influence of significant p-values on the perceived importance of study results
Mohit Bhandari1, Victor M Montori, Emil H Schemitsch
1Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada. bhandari@sympatico.ca
Background:
Statistically significant differences between treatments (i.e., results typically associated with p < 0.05) may not always correspond to important differences upon which to base orthopedic practice. If the hypothesis that p < 0.05 unduly influences the perception of importance of study results were true, we would expect that presenting such a p-value would lead to 1) greater agreement among clinicians about the importance of a study result, and 2) greater perceived importance of a study result, when compared with presenting the same results omitting the p-value.
Methods:
The participants were 3 orthopedics residents, 5 fellows, and 4 attending orthopedic surgeons at a university hospital. We constructed a 40-item questionnaire with the comparison groups, primary outcome of interest, and the results from each of 40 studies. These studies represent a variety of interventions across orthopedic surgery assessed in 2-group comparative intervention studies (randomized trials, observational studies) and were published between 2000 and 2002 in the Journal of Bone and Joint Surgery--American Volume. For each question, respondents were asked to rate the importance of the study results. Participants answered the questionnaire first without p-values and then, 8 weeks later, with p-values (and a random sample of items without p-values). An intra-class correlation quantified agreement between clinicians when answering items with and without p-values. The difference in mean importance scores between the two presentations was also estimated.
Results:
Of 40 eligible clinical comparative studies, 30 reported p < 0.05 for their primary comparison. Without presenting p-values, overall agreement regarding clinical significance among reviewers was fair (ICC = 0.35, 95% CI: 0.25-0.49). In the 30 studies with p < 0.05, mean importance scores (1 = low; 3 = high) were greater when p-values were presented (difference 0.6, CI 0.1-1.1). 10 of 12 reviewers perceived results to be more important when presented with significant p-values.
Interpretation:
When significant, p-values unduly influence the perception of clinicians regarding the importance of study results.
Related Concept Videos
01:10P-value
P-value stands for the probability value. P-value is the probability that, if the null hypothesis is true, the results from another randomly selected sample will be as extreme or more extreme as the results obtained from the given sample.
A large P-value calculated from the data indicates to not reject the null hypothesis. But a higher P-value does not mean that the null hypothesis is true. The smaller the P-value, the more unlikely...
01:37Statistical Significance
01:09Decision Making: P-value Method
First, a specific claim about the population parameter is proposed. The claim is based on the research question and is stated in a simple form. Further, an opposing statement to the claim is also stated. These statements can act as null and alternative hypotheses: a null hypothesis would be a neutral statement while the alternative hypothesis can have a...
01:04Significance Testing: Overview
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
01:10Bonferroni Test
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...