Don't fear peculiar activation functions: EUAF and beyond.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Super-expressive activation functions offer theoretical advantages in approximating complex functions.
• Existing super-expressive functions face challenges in identification, scalability, and practical application.
• Limited understanding of their broad applicability and peculiar forms hinders real-world adoption.

Purpose of the Study:

• To propose a novel super-expressive activation function, the Parametric Elementary Universal Activation Function (PEUAF).
• To demonstrate the effectiveness and generalizability of PEUAF through comprehensive experiments.
• To address key bottlenecks in the development and application of super-expressive activation functions.

Main Methods:

• Development of the Parametric Elementary Universal Activation Function (PEUAF).
• Systematic and comprehensive experimental evaluation on industrial and image datasets (CIFAR-10, Tiny-ImageNet, ImageNet).
• Analysis of models utilizing PEUAF, including mixed activation function configurations.

Main Results:

• PEUAF-based models achieved top performance on several baseline industrial datasets.
• Models incorporating PEUAF in mixed activation settings showed competitive test accuracy on image datasets.
• Generalization of super-expressive activation functions, proving any continuous function can be approximated.

Conclusions:

• PEUAF is an effective and practical super-expressive activation function.
• The study overcomes limitations in identifying and applying super-expressive functions.
• PEUAF contributes to advancing the development and real-world applicability of super-expressive activation functions in deep learning.