Fast and Informative Model Selection Using Learning Curve Cross-Validation.

Area of Science:

• Machine Learning
• Computational Statistics
• Data Science

Background:

• Traditional cross-validation (CV) methods like k-fold and Monte Carlo CV are computationally intensive for large datasets.
• These methods offer limited insight into the learning process of algorithms.
• There is a need for efficient and informative model validation techniques.

Purpose of the Study:

• Introduce a novel validation approach, learning curve cross-validation (LCCV).
• Address the speed and insight limitations of conventional CV methods.
• Provide a more efficient and informative model validation strategy.

Main Methods:

• LCCV iteratively increases training data size, unlike traditional CV's fixed splits.
• It incorporates model selection by discarding non-competitive models early.
• The approach was experimentally validated against standard CV and other methods like successive halving.

Main Results:

• LCCV achieved comparable performance to 5/10-fold CV in over 90% of 75 tested datasets.
• Runtime reductions were substantial, with a median decrease of over 50%.
• Performance deviation from traditional CV was minimal (≤2.5%).

Conclusions:

• LCCV is a computationally efficient and effective alternative to traditional CV.
• The method provides valuable insights into the learning process and data acquisition benefits.
• LCCV offers a promising approach for scalable and informative model validation.