A Theoretical Insight Into the Effect of Loss Function for Deep Semantic-Preserving Learning

Area of Science:

• Machine Learning
• Deep Learning Theory

Background:

• Generalization performance is crucial for machine learning algorithms.
• The impact of loss functions on generalization in deep neural networks (DNNs) is theoretically investigated.

Purpose of the Study:

• To theoretically prove that loss function choice affects DNN generalization performance.
• To provide a stability-based framework for comparing generalization error bounds relative to loss functions.

Main Methods:

• Utilizing the uniform stability concept to analyze generalization error bounds.
• Applying the label distribution learning (LDL) framework for semantically correlated classes.
• Proposing and theoretically analyzing two novel loss functions for LDL.

Main Results:

• Effective loss functions enhance the stability of stochastic gradient descent (SGD).
• Improved stability leads to tighter generalization error bounds and better performance.
• Novel LDL loss functions demonstrate stronger stability compared to existing ones.

Conclusions:

• The choice of loss function significantly impacts DNN generalization.
• The proposed novel loss functions offer improved stability and performance for LDL tasks.
• Theoretical insights are validated by experimental results in facial age, head pose, and image aesthetic assessment.