Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Automated Machine Learning (AutoML) solutions for Deep Learning (DL) have proliferated but lacked standardized, fair comparisons.
• Existing AutoML methods often require significant computational resources and time, limiting practical application.

Purpose of the Study:

• To report the results and analyses of the ChaLearn AutoDL challenge series.
• To establish a benchmark for fair comparison of diverse AutoML solutions across various data modalities.
• To identify practical and efficient AutoML strategies for Deep Learning.

Main Methods:

• Standardized tensor formatting for diverse data types (time series, images, videos, text, tabular).
• Execution of code submissions on hidden tasks with strict time and computational limits.
• Analysis of winning solutions, focusing on modularity, meta-learning, and ensembling.

Main Results:

• Deep Learning methods dominated performance under resource constraints.
• Popular Neural Architecture Search (NAS) methods were impractical in the challenge setting.
• Fine-tuned pre-trained networks with modality-specific architectures were highly effective.
• A modular organization (meta-learner, data ingestor, model selector, learner, evaluator) emerged as crucial.
• Ablation studies highlighted the importance of meta-learning, ensembling, and efficient data management.

Conclusions:

• The challenge established a lasting benchmark and open-sourced code for AutoML research.
• Practical AutoML solutions prioritize modularity, efficient data handling, and leveraging pre-trained models.
• Meta-learning and ensembling significantly contribute to the performance of AutoML systems.