A practical evaluation of AutoML tools for binary, multiclass, and multilabel classification.

Area of Science:

• Machine Learning
• Data Science
• Artificial Intelligence

Background:

• Automated Machine Learning (AutoML) tools simplify model selection for classification tasks.
• A wide array of AutoML frameworks with varying capabilities complicates optimal tool selection.
• Previous benchmarks often focused on limited classification types or fewer tools.

Purpose of the Study:

• To systematically benchmark sixteen AutoML tools across binary, multiclass, and multilabel classification.
• To provide a unified evaluation, unlike prior studies with limited scope.
• To offer insights into tool performance, training times, and suitability for different classification scenarios.

Main Methods:

• Benchmarking sixteen AutoML tools (AutoGluon, AutoSklearn, TPOT, PyCaret, Lightwood, etc.) on 21 real-world datasets.
• Evaluating performance across binary, multiclass, and multilabel classification tasks.
• Conducting feature-based comparisons, time-constrained experiments, and multi-tier statistical validation.

Main Results:

• AutoSklearn demonstrated superior predictive performance in binary and multiclass tasks but required longer training times.
• Lightwood and AutoKeras provided faster training but with reduced predictive accuracy on complex datasets.
• AutoGluon emerged as the top performer, balancing predictive accuracy with computational efficiency across all classification types.

Conclusions:

• Significant performance variations exist among AutoML tools, emphasizing accuracy-speed trade-offs.
• Tool selection should align with specific problem characteristics and resource constraints.
• Several tools exhibit limitations in robust multilabel classification capabilities, necessitating careful consideration.