Selecting fitted models under epistemic uncertainty using a stochastic process on quantile functions.

Area of Science:

• Scientific modeling
• Machine learning
• Data analysis

Background:

• Model selection is vital in science, but complex models present challenges when multiple fit data equally well.
• The primary scientific goal is replicability, requiring models to perform consistently across experiments and labs.
• Existing model selection criteria may not adequately address variations inherent in replication processes.

Purpose of the Study:

• To develop a robust nonparametric method for estimating uncertainty in model empirical risk.
• To ensure model selection prioritizes reproducibility and is insensitive to non-stationary variations.
• To provide a reliable criterion for choosing models when multiple options exhibit similar performance.

Main Methods:

• Developed a nonparametric approach to estimate uncertainty on a model's empirical risk.
• Focused on scenarios with non-stationary replications to ensure robustness.
• Applied the method to both structurally distinct and parameter-variant models.

Main Results:

• The proposed method ensures a model is rejected only when another is reproducibly better.
• Demonstrated favorable comparisons against existing model selection criteria in contexts of replicability.
• Showcased satisfactory performance with large experimental datasets.

Conclusions:

• The new method enhances model selection by focusing on epistemic uncertainty and replicability.
• It offers a more reliable approach than traditional criteria, especially for complex machine learning models.
• This technique improves the scientific practice of fitting models to data, prioritizing consistent performance.