Universal activation function for machine learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Activation functions are crucial components in neural networks, significantly impacting model performance.
• Existing activation functions are often task-specific, requiring manual selection and tuning.
• Optimizing activation functions for diverse problems remains a challenge in deep learning.

Purpose of the Study:

• To introduce a Universal Activation Function (UAF) capable of adapting to various machine learning tasks.
• To demonstrate the UAF's ability to evolve into optimal or near-optimal activation functions for specific problems.
• To evaluate the UAF's performance across quantification, classification, and reinforcement learning domains.

Main Methods:

• The UAF was designed with tunable parameters that gradient descent algorithms can optimize.
• The UAF was tested on diverse benchmark datasets including CIFAR-10, CORA, simulated gas mixtures, ZINC molecular solubility, and the BipedalWalker-v2 reinforcement learning environment.
• Performance was evaluated using metrics such as accuracy, root mean square error (RMSE), and reward accumulation.

Main Results:

• For CIFAR-10 classification, UAF converged to a Mish-like function, achieving near-optimal performance.
• On the CORA dataset and gas mixture quantification, UAF evolved to the identity function, yielding excellent results.
• In molecular solubility prediction and reinforcement learning, UAF adapted to hybrid functions, demonstrating rapid convergence and high rewards.

Conclusions:

• The Universal Activation Function (UAF) offers a flexible and high-performing alternative to traditional activation functions.
• UAF's adaptive nature simplifies model development by automating the selection and optimization of activation functions.
• The UAF demonstrates broad applicability and effectiveness across a spectrum of machine learning tasks.