Self-Growing Binary Activation Network: A Novel Deep Learning Model With Dynamic Architecture

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Network architecture is critical for deep learning model performance, yet designing optimal architectures is challenging and often relies on experience.
• Inappropriate architectures can lead to performance degradation or parameter redundancy, necessitating more efficient design methodologies.

Purpose of the Study:

• To propose a novel deep learning model with a dynamic architecture, the self-growing binary activation network (SGBAN), for progressive network extension.
• To achieve a more compact architecture with higher performance compared to traditional fully connected networks (FCNs) and neural architecture search methods.

Main Methods:

• The self-growing binary activation network (SGBAN) progressively extends fully connected network (FCN) designs.
• Employs function-preserving transformations for architecture expansion and information integration without forgetting prior knowledge.
• Utilizes a novel training technique that is more efficient than training numerous networks in neural architecture search.

Main Results:

• SGBAN demonstrates competitive accuracy against FCNs with identical architectures, validating its optimization capabilities.
• The generated SGBAN architecture on MNIST achieved a 0.59% accuracy improvement with 33.44% fewer parameters than manually designed FCNs.
• Replacing fully connected layers in VGG-19 with SGBAN yielded improved performance with fewer parameters.
• SGBAN outperformed established incremental learning methods on Disjoint MNIST and Disjoint CIFAR-10 tasks.

Conclusions:

• SGBAN offers an effective and efficient method for dynamic deep learning model architecture design.
• The model achieves superior performance, parameter efficiency, and incremental learning capabilities.
• SGBAN presents a promising alternative to manual architecture design and conventional neural architecture search.