A New Correntropy-Based Conjugate Gradient Backpropagation Algorithm for Improving Training in Neural Networks.

Area of Science:

• Machine Learning
• Neural Networks
• Information Theory

Background:

• Mean Square Error (MSE) is a standard criterion for training neural networks.
• Existing methods struggle in non-Gaussian environments and with impulsive or heavy-tailed noise.
• Robustness in neural network training is crucial for real-world applications.

Purpose of the Study:

• To propose novel robust information-theoretic backpropagation (BP) methods.
• To introduce correntropy-based conjugate gradient BP (CCG-BP) algorithms.
• To analyze the convergence properties of the proposed CCG-BP methods.

Main Methods:

• Development of correntropy-based conjugate gradient BP (CCG-BP) algorithms.
• Comparative analysis against common correntropy-based BP and MSE-based CG-BP algorithms.
• Convergence analysis of the novel CCG-BP methods.

Main Results:

• CCG-BP algorithms demonstrate faster convergence than standard correntropy-based BP.
• CCG-BP shows superior performance compared to MSE-based CG-BP, particularly in non-Gaussian settings.
• The proposed method exhibits enhanced robustness against impulsive and heavy-tailed noise, especially at low Signal-to-Noise Ratios (SNR).

Conclusions:

• The novel CCG-BP methods offer a more robust alternative to MSE-based training for neural networks.
• CCG-BP is particularly effective in challenging environments with noise and non-Gaussian distributions.
• The findings are validated through numerical results in function approximation, synthetic function estimation, and chaotic time series prediction.