The Statistical Fragility of Randomized Controlled Trials in Kidney Stone Management: An Analysis of American...

To evaluate the statistical fragility of randomized controlled trials (RCTs) outcomes cited by the American Urological Association (AUA) and European Association of Urology (EAU) nephrolithiasis management guidelines using the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ).

Two-armed RCTs with at least one dichotomous outcome cited by the AUA and EAU were identified. FI was calculated using a 2 × 2 contingency table, adjusting events until statistical significance was lost (P ≥.05). rFI was determined by modifying nonsignificant outcomes until significance was achieved (P <.05). FQ was calculated by dividing FI by sample size.

A total of 1210 outcomes from 138 RCTs were analyzed. The median FI for AUA-cited studies was 4 (IQR 3-6) with an FQ of 0.05 (0.03-0.09), while for EAU-cited studies, median FI was 5 (3-7) with an FQ of 0.04 (0.02-0.06). Over 62% of outcomes required fewer than five event reversals to alter significance. No significant difference in FI (P = .375) or FQ (P = .080) was found between AUA and EAU studies. With respect to median publication year, FI significantly increased in EAU studies published after 2015 (P = .001), and FQ in AUA studies was significantly higher after 2006 (P = .004). In 11.8% of AUA and 14.2% of EAU outcomes, loss to follow-up exceeded the FI.

RCTs cited in AUA and EAU nephrolithiasis guidelines demonstrate statistical fragility. The findings underscore the limitations of P-values as a sole measure of evidence strength. Future research should incorporate FI, rFI, and FQ to enhance clinical decision-making.