Measuring deep learning performance - an empirical study of performance distributions across architectures and tasks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Non-determinism in deep learning algorithms causes performance variations, resulting in performance distributions rather than single values.
• These performance distributions are critical for model robustness but remain underexplored in current research.

Purpose of the Study:

• To investigate the robustness of deep learning model performance to non-determinism across different architectures and tasks.
• To analyze how performance distributions vary and quantify robustness using spread, symmetry, and tail risk metrics.

Main Methods:

• Conducted 186 experiments on image classification (ResNet, ViT) and time series forecasting (Autoformer, iTransformer, NLinear, TSMixer) architectures.
• Ran each experiment 100 times with different random seeds, generating 18,600 total runs to create performance distributions.

Main Results:

• Performance distributions are often non-Gaussian, particularly in time series forecasting.
• Time series models exhibit significantly higher tail risk (underperforming outliers) compared to image classification models.
• Model size and training duration do not systematically predict robustness; early stopping can balance performance and robustness.

Conclusions:

• Mean performance is an insufficient metric for model selection; distributional analysis (spread, symmetry, tail risk) is essential for reliable assessment.
• High tail risk in time series forecasting poses challenges for Trustworthy AI in critical applications.
• Understanding performance distributions is vital for developing robust and reliable deep learning models.