RoSwish: A novel Rotating Swish activation function with adaptive rotation around zero.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Existing activation functions often have limitations in nonlinear expressive capabilities.
• A need exists for activation functions that offer improved smoothness, gradient stability, and adaptability.
• Current models struggle with representing non-stationary signals effectively.

Purpose of the Study:

• To introduce the novel Rotating Swish (RoSwish) activation function.
• To enhance nonlinear expressive capabilities and performance in neural networks.
• To explore activation functions inspired by non-stationary stochastic process theory.

Main Methods:

• Developed the RoSwish activation function: f(x)=(x+α)·sigmoid(β·x)-0.5·α, featuring learnable parameters α and β.
• Integrated advantageous features from Rectified Linear Unit (ReLU), Gaussian Error Linear Unit (GELU), Exponential Linear Unit (ELU), Parametric ReLU (PReLU), and Swish.
• Designed a series of activation functions based on non-stationary sine waves.

Main Results:

• RoSwish achieved significant performance improvements in MNIST classification (≥16.02%), MNIST autoencoding (MSE reduction ≥5%), tweet tagging (≥3.65%), and ETTm2 time series prediction (≥6.06%).
• Combined RoSwish with batch normalization for a 30.19% improvement in tweet tagging under low learning rates.
• The non-stationary sine wave-based activation functions improved MNIST Classification performance by at least 6.55%.

Conclusions:

• RoSwish offers superior nonlinear expressive capabilities and adaptability compared to existing activation functions.
• The proposed activation functions show promise for enhancing neural network performance across diverse applications.
• The study provides a new avenue for developing advanced activation functions for complex data representation.