Kafnets: Kernel-based non-parametric activation functions for neural networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural networks typically use fixed nonlinear activation functions, limiting model flexibility.
• Existing methods for adaptable activation functions have not achieved widespread adoption.
• Research into more flexible activation functions remains an open area.

Purpose of the Study:

• To introduce a novel family of flexible activation functions: kernel activation functions (KAFs).
• To explore variations in designing and initializing KAFs, including multidimensional schemes.
• To provide an overview of alternative adaptable activation function techniques.

Main Methods:

• Developed flexible activation functions based on kernel expansions at each neuron.
• Proposed multiple design and initialization strategies for KAFs.
• Investigated multidimensional KAFs for nonlinear information combination.

Main Results:

• KAFs can approximate any convex or non-convex mapping over a subset of real numbers.
• KAFs are smooth, linear in parameters, and amenable to regularization (e.g., L1 for sparsity).
• KAFs satisfy properties not simultaneously met by other known models.

Conclusions:

• The proposed kernel activation functions offer significant flexibility and desirable mathematical properties.
• KAFs represent a promising advancement in neural network activation function design.
• Experimental validation supports the efficacy of the proposed KAFs.