A regression-based equivalence test for model validation: shifting the burden of proof

Area of Science:

• Statistical modeling
• Scientific methodology

Background:

• Model validation typically compares predictions to independent observations.
• Traditional statistical tests are designed to detect differences, not similarities, and are less effective for model validation.
• Existing equivalence tests for model validation primarily focus on comparing means.

Purpose of the Study:

• To present an alternative model validation strategy using regression and statistical tests of equivalence.
• To develop a test that assesses similarity between individual predictions and observations, not just means.
• To provide a formal and superior method for model validation.

Main Methods:

• Utilized regression analysis and statistical tests of equivalence for model validation.
• Implemented equivalence tests that reverse the null hypothesis, positing populations are different and seeking data to prove otherwise.
• Applied the strategy to three case studies with diverse modeling objectives and sample sizes.

Main Results:

• The proposed equivalence testing strategy demonstrated effectiveness in model validation across varied scenarios.
• The new method successfully validated models by assessing similarity between individual predictions and observations.
• The strategy proved superior to traditional statistical tests in all demonstrated case studies.

Conclusions:

• The proposed strategy offers a formal and robust approach to model validation.
• Equivalence testing provides a more appropriate framework for assessing model predictive accuracy than traditional difference-testing methods.
• This method enhances the ability to demonstrate model utility and reliability through rigorous validation.