An extensive experimental survey of regression methods.

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Regression is a fundamental machine learning task with numerous algorithmic approaches.
• Evaluating and comparing these diverse regression models is crucial for practical applications.
• The UCI machine learning repository provides a comprehensive benchmark for regression algorithms.

Purpose of the Study:

• To conduct a large-scale comparative analysis of 77 popular regression models.
• To identify high-performing and efficient regression models across various datasets.
• To provide guidance on selecting appropriate regression models for new problems.

Main Methods:

• Evaluation of 77 regression models from 19 families.
• Utilized 83 regression datasets from the UCI machine learning repository.
• Performance metrics included squared correlation (R²), speed, and memory usage.

Main Results:

• Cubist, Gradient Boosted Machine (gbm), Boosting Ensemble (bstTree), and M5 regression trees emerged as top performers.
• Cubist achieved the best R² on 15.7% of datasets, with 89.1% within 0.2 difference.
• Extremely Randomized Trees (extraTrees) achieved the best R² on 33.7% of datasets.
• Least Angle Regression (lars) was the fastest model, while Non-Negative Least Squares (nnls) required the least memory.

Conclusions:

• For new regression problems, models within the top-10 consistently achieve R² values close to the best attainable.
• Model selection should consider a trade-off between performance (e.g., R²), speed, and computational resources.
• The study provides empirical evidence to guide the selection of effective regression models.