Double Oracle Neural Architecture Search for Game Theoretic Deep Learning Models.

Area of Science:

• Deep Learning
• Game Theory
• Machine Learning

Background:

• Training deep learning models, including Generative Adversarial Networks (GANs) and Adversarial Training (AT), is complex due to challenges in finding Nash equilibria in large strategy spaces.
• Existing methods often struggle with scalability and convergence in these complex training scenarios.

Purpose of the Study:

• To propose a new approach for training deep learning models using game theory concepts, specifically a double-oracle framework.
• To extend the double-oracle framework to Adversarial Neural Architecture Search (NAS) for GANs and AT algorithms.
• To improve the performance and robustness of GANs and AT models.

Main Methods:

• Developed a double-oracle framework utilizing best response oracles, generalizing player strategies as trained models.
• Employed linear programming to compute meta-strategies and pruned weakly-dominated strategies for scalability.
• Applied the framework to develop DONAS-GAN and DONAS-AT algorithms.

Main Results:

• Experiments on MNIST, CIFAR-10, and TinyImageNet demonstrated significant improvements for DONAS-GAN.
• Evaluations on CIFAR-10, SVHN, and TinyImageNet showed enhanced robustness of DONAS-AT under FGSM and PGD attacks.
• All variants showed substantial improvements in both qualitative and quantitative metrics compared to base architectures.

Conclusions:

• The proposed double-oracle framework effectively enhances the training of GANs and AT models.
• The DONAS-GAN and DONAS-AT variants offer superior performance and robustness.
• This game theory-based approach provides a scalable and effective solution for complex deep learning training challenges.