The Robustness Index: Going Beyond Statistical Significance by Quantifying Fragility.

Area of Science:

• Biostatistics
• Research Methodology
• Scientific Reproducibility

Background:

• Statistical significance is a common but limited metric for evaluating research findings, particularly concerning reproducibility.
• Existing measures of statistical fragility, such as the unit fragility index and fragility quotient, are dependent on sample size and single unit changes.
• These limitations hinder the accurate assessment of research robustness against assumption violations.

Purpose of the Study:

• To introduce a novel metric, the Robustness Index (RI), for quantifying statistical fragility.
• To develop a measure of research robustness that is independent of sample size.
• To provide a more reliable and interpretable assessment of the stability of statistical findings.

Main Methods:

• The Robustness Index (RI) quantifies how changes in sample size impact statistical significance.
• For non-significant findings, the RI is the factor by which sample size must be multiplied to achieve significance.
• For significant findings, the RI is the factor by which sample size must be divided to lose significance.

Main Results:

• The RI offers a measure of statistical fragility that is independent of sample size.
• Higher RI values indicate greater robustness for both significant and non-significant research findings.
• The RI provides a simple and interpretable metric for assessing the stability of statistical results.

Conclusions:

• The Robustness Index (RI) addresses limitations of existing fragility measures by decoupling fragility from sample size.
• The RI facilitates cross-study comparisons and has the potential to increase confidence in biomedical research outcomes.
• This metric can enhance the evaluation of research findings and contribute to improved scientific reproducibility.