Bayesian Optimization for Sparse Neural Networks With Trainable Activation Functions.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Activation functions are crucial for neural network performance.
• Trainable activation functions show promise in improving network accuracy and reducing overfitting.
• Existing methods often lack automatic parameter estimation during the learning process.

Purpose of the Study:

• To propose a novel trainable activation function for deep neural networks.
• To develop a fully Bayesian model for estimating both network weights and activation function parameters.
• To enhance neural network performance, particularly in reducing overfitting and improving convergence time.

Main Methods:

• A fully Bayesian model was developed for parameter estimation.
• Markov Chain Monte Carlo (MCMC)-based optimization was employed for inference.
• The proposed activation function and Bayesian estimation were tested on diverse datasets and CNN architectures.

Main Results:

• The proposed trainable activation function demonstrated improved model accuracy.
• Bayesian estimation of parameters effectively reduced overfitting.
• The efficient sampling scheme ensured convergence to the global maximum, improving convergence time.
• Successful implementation across various Convolutional Neural Network (CNN) architectures and tasks (classification, regression).

Conclusions:

• The proposed trainable activation function, coupled with Bayesian parameter estimation, offers a versatile and effective approach to enhance deep neural network performance.
• The method successfully addresses challenges related to parameter estimation and overfitting.
• The approach shows significant potential for improving accuracy and convergence in deep learning models.