Nonmonotonic generalization bias of Gaussian mixture models

Area of Science:

• Machine Learning
• Statistical Physics
• Information Theory

Background:

• Learning theories posit generalization bias increases with adaptive parameters.
• Generalization bias is the difference between training and generalization error.
• This relationship is generally observed across various models.

Purpose of the Study:

• To investigate the behavior of generalization bias in Gaussian mixture models.
• To examine the impact of symmetry breaking on learning and generalization.
• To challenge the conventional understanding of the generalization bias-parameter relationship.

Main Methods:

• Theoretical analysis of a Gaussian mixture model.
• Computation of the neural information criterion's dependence on temperature.
• Numerical cross-validation experiments to confirm findings.

Main Results:

• Observed a violation of the general tendency for generalization bias to increase with parameters.
• Found that the effective number of adaptive parameters increases below the symmetry breaking point.
• Demonstrated a decrease in generalization bias in this specific regime.

Conclusions:

• The study highlights a counterintuitive finding in learning and generalization for Gaussian mixture models.
• Symmetry breaking points can lead to decreased generalization bias despite increased effective parameters.
• Results necessitate a refinement of current theories on learning and generalization bias.