Controlling the error probabilities of model selection information criteria using bootstrapping.

Area of Science:

• Statistics
• Computational Statistics
• Econometrics

Background:

• Information criteria like AIC are widely used for comparing non-nested statistical models.
• Current information-theoretic model selection lacks explicit error rate control, unlike classical hypothesis testing.
• Extending error control concepts to multi-model selection without a null hypothesis is a significant challenge.

Purpose of the Study:

• To extend the concepts of Type-I and Type-II errors to scenarios involving more than two models.
• To introduce the Error Control for Information Criteria (ECIC) method for robust model selection.
• To provide a bootstrap-based approach for controlling Type-I error rates in information-theoretic model selection.

Main Methods:

• Extension of Type-I and Type-II error definitions to multi-model comparisons.
• Development of the Error Control for Information Criteria (ECIC) method.
• Utilizing bootstrap resampling and Difference of Goodness of Fit (DGOF) distributions for error control.

Main Results:

• Demonstrated the feasibility of extending error control to information criteria.
• ECIC method successfully controls Type-I error rates in model selection.
• The method shows value in time series and regression analyses with simulated and empirical data.

Conclusions:

• The ECIC method offers a novel approach to enhance the reliability of model selection using information criteria.
• ECIC provides explicit control over Type-I error rates, a crucial feature for statistical inference.
• The publicly available R package facilitates the application of ECIC in diverse research areas.