What Are the Implications of Alternative Alpha Thresholds for Hypothesis Testing in Orthopaedics?

Clinical research in orthopaedics typically reports the presence of an association after rejecting a null hypothesis of no association using an alpha threshold of 0.05 at which to evaluate a calculated p value. This arbitrary value is a factor that results in the current difficulties reproducing research findings. A proposal is gaining attention to lower the alpha threshold to 0.005. However, it is currently unknown how alpha thresholds are used in orthopaedics and the distribution of p values reported.

We sought to describe the use of alpha thresholds in two orthopaedic journals by asking (1) How frequently are alpha threshold values reported? (2) How frequently are power calculations reported? (3) How frequently are p values between 0.005 and 0.05 reported for the main hypothesis? (4) Are p values less than 0.005 associated with study characteristics such as design and reporting power calculations?

The 100 most recent original clinical research articles from two leading orthopaedic journals at the time of this proposal were reviewed. For studies without a specified primary hypothesis, a main hypothesis was selected that was most consistent with the title and abstract. The p value for the main hypothesis and lowest p value for each study were recorded. Study characteristics including details of alpha thresholds, beta, and p values were recorded. Associations between study characteristics and p values were described. Of the 200 articles (100 from each journal), 23 were randomized controlled trials, 141 were cohort studies or case series (defined as a study in which authors had access to original data collected for the study purpose), 31 were database studies, and five were classified as other.

An alpha threshold was reported in 166 articles (83%) with all but two reporting a value 0.05. Forty-two articles (21%) reported performing a power calculation. The p value for the main hypothesis was less than 0.005 for 88 articles (44%), between 0.05 and 0.005 for 67 (34%), and greater than 0.05 for 29 (15%). The smallest p value was between 0.05 and 0.005 for 39 articles (20%), less than 0.005 for 143 (72%), and either not provided or greater than 0.05 for 18 (9%). Although 50% (65 of 130) cohort and database papers had a main hypothesis p value less than 0.005, only 26% (6 of 23) randomized controlled trials did. Only 36% (15 of 42) articles reporting a power calculation had a p value less than 0.005 compared with 51% (73 of 142) that did not report one.

Although a lower alpha threshold may theoretically increase the reproducibility of research findings across orthopaedics, this would preferentially select findings from lower-quality studies or increase the burden on higher quality ones. A more-nuanced approach could be to consider alpha thresholds specific to study characteristics. For example, randomized controlled trials with a prespecified primary hypothesis may still be best evaluated at 0.05 while database studies with an abundance of statistical tests may be best evaluated at a threshold even below 0.005.

Surgeons and scientists in orthopaedics should understand that the default alpha threshold of 0.05 represents an arbitrary value that could be lowered to help reduce type-I errors; however, it must also be appreciated that such a change could increase type-II errors, increase resource utilization, and preferentially select findings from lower-quality studies.