Empirical strategy for stretching probability distribution in neural-network-based regression.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Statistical Modeling

Background:

• Prediction performance in artificial neural networks (ANNs) relies heavily on optimal weight determination between layers.
• The choice of loss function significantly impacts ANN performance during back-propagation and gradient descent optimization.
• Existing loss functions may not adequately address distribution inconsistencies between predicted values and true labels.

Purpose of the Study:

• To propose a novel loss function, weighted empirical stretching (WES), designed to enhance prediction accuracy in ANNs.
• To address distribution error by increasing the overlap between predicted and label distributions.
• To develop a loss function applicable to diverse label distribution shapes and robust to noise.

Main Methods:

• Introduced weighted empirical stretching (WES), a loss function measuring distribution error as the inconsistency between predicted and label distributions.
• WES aims to maximize the overlap area of these distributions, incorporating a scaling hyperparameter (β).
• Tested WES using feedforward neural networks with Fourier-extracted data from various ideal label distributions (unimodal, skewed unimodal, bimodal, skewed bimodal) under different noise levels.

Main Results:

• Weighted empirical stretching (WES) generally outperformed commonly used loss functions in regression tasks.
• The proposed WES loss function demonstrated robustness across various noise levels.
• Significant improvements in Root Mean Square Error (RMSE) were observed in the extreme domains (tail regions) of the distribution.

Conclusions:

• WES offers a superior approach to loss function design in ANNs by focusing on distribution overlap.
• The method's effectiveness across diverse distributions and noise levels highlights its versatility.
• The enhanced performance in predicting extreme values suggests potential applications in forecasting abnormal events in complex systems like natural disasters and financial markets.