Continuously Constructive Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Traditional deep learning manually selects network architecture through trial and error.
• Optimizing network architecture is crucial for effective deep learning model performance.

Purpose of the Study:

• To propose two novel methods for automatically updating deep learning network structures concurrently with weight learning.
• To introduce a new parameterization that encapsulates network depth and complexity within control parameters.

Main Methods:

• Developed 'tunnel networks' that adjust complexity at the hidden unit level.
• Developed 'budding perceptrons' that adjust complexity at the network layer level.
• Both methods utilize control parameters to dynamically add complexity (units or layers).

Main Results:

• Demonstrated effectiveness on synthetic (two-spiral) and real-world datasets (MNIST, MIRFLICKR, CIFAR).
• The methods successfully adjusted network complexity to match task complexity using a consistent set of hyperparameters.
• Achieved comparable or improved performance by automating architecture search.

Conclusions:

• The proposed methods offer an automated alternative to manual architecture design in deep learning.
• Dynamic adjustment of network complexity enhances model adaptability and efficiency.
• This approach simplifies the deep learning workflow by integrating architecture and weight optimization.