Errors in Statistical Inference Under Model Misspecification: Evidence, Hypothesis Testing, and AIC | JoVE Visualize

Area of Science:

Statistical Inference
Ecological Modeling
Model Selection

Background:

Statistical inference methods, including Neyman-Pearson hypothesis testing, Fisher significance testing, information criteria, and evidential statistics, have diverse applications but lack comprehensive comparison.
Evidential statistics, implemented as evidence functions, estimate the relative distance of models from the data-generating process, offering a novel approach to statistical inference.

Purpose of the Study:

To analytically and numerically compare the performance of Neyman-Pearson hypothesis testing, Fisher significance testing, information criteria, and evidential statistics.
To evaluate error rates, particularly Type 1 and Type 2 errors, under both correct and misspecified model conditions.
To assess the interpretability and practical utility of these statistical approaches, especially within ecological modeling contexts.

Main Methods:

Analytical and numerical approximations were used to evaluate the performance of different statistical inference methods.
Focus was placed on quantifying error frequencies (e.g., Type 1 and Type 2 errors) for each method.
Model misspecification scenarios were investigated to assess method robustness.

Main Results:

Evidential analysis demonstrates error probabilities that decrease to zero with increasing sample size, even under model misspecification.
Neyman-Pearson testing exhibits significant performance degradation under misspecification, with error rates potentially increasing and approaching 1.
Consistent information criteria function as evidence functions, while MSE-minimizing criteria (e.g., AIC) do not consistently exhibit desirable error properties.

Conclusions:

Evidential analysis, using evidence functions, provides a statistically sound and intuitive framework for model selection, ensuring error probabilities approach zero.
Evidential analysis maintains desirable properties like a higher probability of selecting the best model, which increases monotonically with sample size, even when models are misspecified.
Evidence functions align with the objectives of model selection in ecology, offering a reliable alternative to traditional hypothesis testing and information criteria.