On joint parameterizations of linear and nonlinear functionals in neural networks.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Traditional neural networks rely on fixed nonlinear activation functions.
• Existing functions like sigmoid and rectified linear units have limitations.
• Rich functional representation is crucial for complex machine learning tasks.

Purpose of the Study:

• To propose a new class of nonlinear operators called rectified parametric sigmoid units.
• To develop a dual learning paradigm optimizing both linear convolutional weights and nonlinear operator parameters.
• To enhance functional representation capabilities in machine learning models.

Main Methods:

• Introduced a novel nonlinear operator class: rectified parametric sigmoid units.
• Developed a dual learning paradigm for joint optimization of linear and nonlinear parameters.
• Incorporated scale, shift, and shape parameters for flexible activation functions.

Main Results:

• Rectified parametric sigmoid units offer a rich functional representation.
• Joint learning of convolutional and rectified parametric sigmoid parameters demonstrated outstanding performance.
• The new class includes the standard rectified linear unit as a limit case.

Conclusions:

• The proposed nonlinear operators and dual learning paradigm significantly advance machine learning.
• Learnable parameters associated with nonlinear operators open new research avenues.
• This approach enhances performance in both shallow and deep learning frameworks.