ConvNets with Smooth Adaptive Activation Functions for Regression.

Area of Science:

• Machine Learning
• Deep Learning
• Computer Vision

Background:

• Adaptive Activation Functions (AAFs) are parameters within Neural Networks (NNs) that control activation function shapes and are trained alongside other network parameters.
• AAFs have previously demonstrated improved performance in Convolutional Neural Networks (CNNs) for various classification tasks.

Purpose of the Study:

• To propose and apply AAFs to CNNs for regression tasks.
• To decrease the bias in regression NNs by applying AAFs in the penultimate layer.
• To address the overfitting issue associated with existing AAFs in regression.

Main Methods:

• Introduction of a Smooth Adaptive Activation Function (SAAF) with a piecewise polynomial form.
• SAAF's ability to approximate any continuous function with arbitrary error degree.
• Ensuring a bounded Lipschitz constant for bounded model parameters within SAAF.

Main Results:

• NNs incorporating SAAF can prevent overfitting through simple regularization of model parameters.
• Empirical evaluation of CNNs utilizing SAAFs.
• Achieved state-of-the-art performance on age and pose estimation datasets.

Conclusions:

• SAAFs offer a robust solution for improving regression performance in CNNs.
• The proposed SAAF effectively mitigates overfitting while maintaining high accuracy.
• This work extends the application of AAFs to regression, yielding superior results.