Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Early automated machine learning (AutoML) focused on traditional ML pipelines and hyperparameter tuning.
• A recent trend in AutoML is neural architecture search for deep learning models.

Purpose of the Study:

• Introduce Auto-PyTorch, a framework for fully automated deep learning (AutoDL).
• Jointly and robustly optimize network architecture and training hyperparameters.
• Achieve state-of-the-art performance on tabular benchmarks.

Main Methods:

• Combine multi-fidelity optimization with portfolio construction for warm-starting and ensembling.
• Utilize deep neural networks (DNNs) and common baselines for tabular data.
• Introduce LCBench, a new benchmark for learning curves in DNNs.
• Conduct extensive ablation studies.

Main Results:

• Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks.
• Demonstrate the effectiveness of joint optimization of architecture and hyperparameters.
• Show superior performance compared to state-of-the-art AutoML competitors.

Conclusions:

• Auto-PyTorch enables fully automated deep learning by integrating architecture search and hyperparameter optimization.
• The framework's approach is robust and achieves top performance on tabular data.
• LCBench provides a valuable resource for studying DNN learning curves.