Don't lose samples to estimation.

Area of Science:

• Machine Learning
• Statistical Modeling

Background:

• Traditional predictive modeling reserves a test set, reducing available data for model training.
• This data loss is particularly problematic in low-sample-size scenarios, compromising model robustness.

Purpose of the Study:

• To propose an alternative methodology for estimating predictive model performance without sacrificing data.
• To address the limitations of traditional test set usage in predictive modeling.

Main Methods:

• Shift focus from evaluating a single model instance to evaluating the entire model-producing pipeline.
• Utilize cross-validation techniques where the pipeline is trained on subsets of the data to estimate performance.
• Incorporate corrections for the 'winner's curse' when comparing multiple pipelines.

Main Results:

• The proposed pipeline-centric evaluation method avoids data loss to estimation, preserving all samples for final model training.
• Performance estimation is achieved by repeatedly training the pipeline on data subsets, providing a more reliable out-of-sample estimate.
• The approach ensures that the final operational model is trained on the maximum available data.

Conclusions:

• Estimating pipeline performance is a more data-efficient strategy than traditional test set holdout, especially for small datasets.
• This methodology enhances the utility of all available data for building the final predictive model.
• Further considerations are needed for selecting among multiple pipelines to avoid biased performance estimates.