Integration between constrained optimization and deep networks: a survey.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep networks are increasingly integrated with optimization techniques.
• Physical and knowledge-based constraints are crucial for practical deep network applications.
• Existing literature shows growing interest in combining constrained optimization with deep learning.

Purpose of the Study:

• To survey and analyze the literature on integrating constrained optimization with deep neural networks.
• To examine how physical constraints (e.g., FLOPS, latency) and knowledge constraints impact network design and training.
• To explore methods for incorporating logic and semantic constraints into deep learning models.

Main Methods:

• Literature review of constrained optimization and deep network integration.
• Analysis of hyper-parameter tuning and neural architecture search (NAS) under constraints.
• Exploration of multi-objective optimization (MOO) and penalty-based approaches in NAS.
• Investigation of logic-neural integration and semantic loss functions.

Main Results:

• Constrained optimization offers a framework to optimize network structure beyond accuracy, considering computational capacity and latency.
• Integrating physical and context-specific knowledge constraints during training enhances deep network performance.
• Neural Architecture Search (NAS) can be framed as a multi-objective optimization problem or solved using loss function penalties.
• Logic-neural integration, particularly through semantic loss, provides a method to enforce output constraints.

Conclusions:

• The integration of constrained optimization with deep networks is a promising research direction.
• Applying constraints during hyper-parameter tuning, NAS, and training leads to more efficient and context-aware deep learning models.
• Future work should focus on developing novel methods for logic-neural integration and semantic loss to further enhance deep network capabilities.