Activation function cyclically switchable convolutional neural network model

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Activation functions are critical hyperparameters significantly impacting neural network training and performance.
• Selecting the optimal activation function for specific tasks remains a challenging problem in deep learning.
• Existing solutions include fixed activation functions or trainable activation functions, each with limitations.

Purpose of the Study:

• To propose a novel neural network architecture that dynamically adapts its activation function during training.
• To introduce the activation function cyclically switchable convolutional neural network (AFCS-CNN) model.
• To demonstrate an alternative to fixed or trainable activation function approaches for improved model performance.

Main Methods:

• Developed the AFCS-CNN model, which cyclically switches between multiple activation functions during the training process.
• The model self-regulates by selecting the most optimal activation function based on performance, adapting to decreasing performance.
• Conducted ablation studies on the Cifar-10 dataset to determine optimal CNN models and hyperparameters for the AFCS-CNN structure.

Main Results:

• The AFCS-CNN model demonstrated state-of-the-art performance across various convolutional neural network (CNN) models.
• Experiments on diverse datasets confirmed the effectiveness and adaptability of the proposed AFCS-CNN structure.
• The model achieved significant success, outperforming conventional approaches in multiple scenarios.

Conclusions:

• The AFCS-CNN model offers a simple yet highly effective method for optimizing activation function selection in neural networks.
• This dynamic switching approach provides a robust alternative to static or trainable activation functions.
• The proposed architecture achieves state-of-the-art results, highlighting its potential for advancing deep learning applications.