Evaluation of multiple prediction models: A novel view on model selection and performance assessment

Area of Science:

• Machine Learning
• Statistical Modeling
• Predictive Analytics

Background:

• Model selection and performance assessment are critical in machine learning for developing predictive models, particularly in medical diagnosis and prognosis.
• Current common practice involves selecting a single best model via cross-validation and evaluating it on an independent dataset.

Purpose of the Study:

• To propose and evaluate a novel strategy for simultaneous assessment of multiple machine learning models.
• To increase the probability of correctly identifying a sufficiently performing model.
• To address the issue of inflated family-wise error rate in model evaluation.

Main Methods:

• Simultaneous evaluation of multiple models (e.g., from varied hyperparameters or algorithms).
• Application of the maxT-approach for multiplicity adjustment to control the family-wise error rate.
• Utilizing the joint distribution of test statistics for performance measures.

Main Results:

• Evaluating only a single final model is suboptimal for prediction tasks.
• Simultaneous evaluation of several promising models (e.g., within one standard error of the best) increases the probability of identifying a good model.
• The proposed strategy enhances final model performance in simulation studies.

Conclusions:

• Simultaneous model evaluation is superior to single-model evaluation for machine learning tasks.
• The maxT-approach effectively controls the family-wise error rate in this context.
• This strategy improves both the identification of high-performing models and overall predictive performance.