Unbreakable? An analysis of the fragility of randomized trials that support diabetes treatment guidelines

To analyze the robustness of randomized controlled trials (RCTs) referenced in the American Diabetes Association's Standards of Medical Care in Diabetes-2017 using the Fragility Index (FI) and Fragility Quotient (FQ).

We performed a systematic survey of all RCTs referenced in the Standards of Medical Care in Diabetes-2017. One investigator screened for trials and then recorded data from them, including sample size per group, event rate per group, and the dichotomous outcome analyzed by trialists. The FI and FQ for each outcome were calculated. Outcomes were then surveyed to determine whether the outcome used for analysis aligned with the guideline recommendation.

Thirty-five RCTs were included in this study. Thirty-three of 35 (94%) FIs were based on the trial outcomes referenced in the clinical practice guideline. The median sample size was 2548 participants ([IQR], 522-6946). The median total number of events for each outcome was 403 (IQR, 86-969). Nineteen (54%) P Values were below 0.05, 8 (22.4%) were below 0.01. The median FI for all trials was 16 (IQR, 4-29). The median FQ was 0.007 (IQR, 0.003-0.014). FI was not related to risk of bias or Science Citation Index but was significantly correlated with sample size (for significant trials; r=0.77, P < .001, for neutral trials; r=0.76, P < .001).

We found that the robustness of RCTs varied, but on the whole were not robust in nature. Most trials demonstrated a modest FI and FQ. As a result, conclusions drawn from these trials should take this information into account.