Related Experiment Video
Updated: Jul 17, 2025
A Reliable and Reproducible Critical-Sized Segmental Femoral Defect Model in Rats Stabilized with a Custom External Fixator
Published on: March 24, 2019
The Fragility of Statistical Significance in Sham Orthopaedic Surgery: A Systematic Review of Randomized Controlled
Christian Pearsall1, Michael Constant, Bryan M Saltzman
1From the Department of Orthopedic Surgery, Columbia University Irving Medical Center, New York, NY (Pearsall, Constant, Levine, and Trofa), the Department of Orthopedic Surgery, OrthoCarolina, Charlotte, NC (Saltzman), and the Department of Orthopedic Surgery, Mount Sinai Health System, New York, NY (Parisien).
Objectives:
The purpose of this study was to determine the stability of statistical findings among sham surgery randomized controlled trials (RCTs) in orthopaedic surgery using fragility analysis.
Methods:
PubMed systematic review was conducted to include studies reporting dichotomous outcomes pertaining to sham surgery. The final review included eight RCTs involving only partial meniscectomies and vertebroplasties from 2009 to 2020. With a fixed sample size with dichotomous outcome measures (events versus non-events), the Total Fragility Index (TFI), which is composed of the fragility index (FI) and reverse fragility index (RFI), was calculated by altering the ratio of events to non-events in an iterative fashion until results were reversed from significant to nonsignificant findings (FI) or vice versa (RFI). The TFI, FI, and RFI were divided by their sample sizes to obtain the respective total fragility quotient, fragility quotient (FQ), and reverse fragility quotient. Median fragility indices and quotients were reported for all studies.
Results:
The eight RCTs included 50 dichotomous outcomes involving either partial meniscectomies or vertebroplasties, with a median TFI and total fragility quotient of 5 [interquartile range (IQR) 4 to 6] and 0.035 (IQR 0.028 to 0.048), respectively, indicating that a median of five total patients or 3.5 per 100 patients would need to experience a different outcome to reverse significant or insignificant findings for each of the eight trials. Among the 8 statistically significant ( P < 0.05) outcome events (16%), the respective FI and FQ were 2 (IQR 1 to 5) and 0.018 (IQR 0.010 to 0.044). Among the 42 statistically insignificant outcome events (84%), the respective RFI and reverse fragility quotient were 5 (IQR 4 to 6) and 0.04 (IQR 0.034 to 0.048). The median number of patients lost to follow-up was 1.5 (IQR 0.5 to 2).
Conclusion:
The unstable findings in partial meniscectomy and vertebroplasty sham surgical RCTs undermine their study conclusions and recommendations. We recommend using fragility analysis in future sham surgical RCTs to contextualize statistical findings.
Level Of Evidence:
Level IV; Systematic Review.
Insights
Statistical findings in sham surgery randomized controlled trials (RCTs) for orthopaedic procedures are often unstable. Fragility analysis reveals that a small number of outcome changes can reverse study results, questioning their reliability.
Area of Science:
- Orthopaedic surgery
- Clinical trial methodology
- Biostatistics
Background:
- Sham surgery randomized controlled trials (RCTs) are used in orthopaedics.
- The statistical stability of findings in these trials is often unexamined.
- Fragility analysis offers a method to assess the robustness of statistical results.
Purpose of the Study:
- To determine the statistical stability of findings in sham surgery RCTs within orthopaedic surgery.
- To apply fragility analysis to evaluate the reliability of reported outcomes.
- To assess the impact of potential outcome reversals on study conclusions.
Main Methods:
- A systematic review of PubMed was conducted for sham surgery RCTs with dichotomous outcomes.
- Eight RCTs involving partial meniscectomies and vertebroplasties (2009-2020) were included.
- Total Fragility Index (TFI), Fragility Index (FI), and Reverse Fragility Index (RFI) were calculated and normalized by sample size.
Main Results:
- A median of 5 patients (3.5 per 100) experiencing different outcomes would reverse the statistical significance of findings across included trials.
- Statistically significant outcomes (16%) had a median FI of 2 (FQ 0.018).
- Statistically insignificant outcomes (84%) had a median RFI of 5 (Reverse Fragility Quotient 0.04).
Conclusions:
- Statistical findings in sham surgery RCTs for partial meniscectomy and vertebroplasty are frequently unstable.
- The fragility of these results undermines the conclusions and recommendations of such studies.
- Fragility analysis should be incorporated into future sham surgery RCTs to contextualize statistical findings.
