Looking for statistical stability: a new method of evaluating reliability of statistical tests

Area of Science:

• Statistics
• Statistical Methods

Background:

• Assessing the reliability of statistical tests is crucial for data analysis.
• Existing methods like statistical power offer insights but may not capture all aspects of reliability.

Purpose of the Study:

• Introduce a novel method, stability calculation, for evaluating the reliability of statistical tests.
• To provide a complementary measure to statistical power for assessing test dependability.

Main Methods:

• Stability calculation is a subsampling technique.
• It involves analyzing results from subsamples of the original data.
• The method quantifies the percentage of data removal that alters the original findings.

Main Results:

• Stability calculation yields a "stability percentage" indicating test reliability.
• Higher stability percentages correlate with more reliable statistical tests.
• The method identifies influential data points within the sample.

Conclusions:

• Stability calculation offers a new, descriptive, non-inferential metric for statistical test reliability.
• It enhances the assessment of statistical tests alongside traditional measures like power.
• The technique provides insights into sample composition and data point influence.