Stochastic Selection of Activation Layers for Convolutional Neural Networks

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning excels in pattern recognition, image segmentation, and classification tasks.
• Activation functions are critical for deep neural network discriminative capabilities.
• Static activation functions treat neurons/layers identically, while dynamic functions learn layer/neuron-specific parameters.

Purpose of the Study:

• To propose a hybrid approach combining static and dynamic activation functions for deep neural networks.
• To enhance model design by stochastically selecting activation functions at each layer.
• To create versatile Convolutional Neural Network (CNN) models for standalone use or ensemble components.

Main Methods:

• A mixture of static and dynamic activation functions is proposed, stochastically selected per layer.
• Model design involves modifying layers using functional blocks from high-performing CNNs.
• Each activation layer (typically ReLU) in a CNN is replaced by a stochastically chosen activation function from a predefined set.

Main Results:

• The proposed method results in CNNs with varied activation function layers.
• This approach aims to balance performance gains of dynamic functions with the computational efficiency of static functions.
• The design facilitates creating novel CNN architectures adaptable for various applications.

Conclusions:

• The stochastic mixture of static and dynamic activation functions offers a promising direction for deep neural network design.
• This hybrid strategy can potentially mitigate overfitting and reduce computational demands associated with purely dynamic functions.
• The developed method enables the creation of flexible and robust CNN models for diverse classification challenges.