Gaussian Mean Field Regularizes by Limiting Learned Information.

Area of Science:

• Machine Learning
• Statistical Inference
• Deep Learning

Background:

• Variational inference with factorized Gaussian posteriors is common for learning model parameters and hidden variables.
• A poorly understood regularizing effect is empirically observed in these methods.

Purpose of the Study:

• To elucidate the mechanism by which mean field inference improves generalization.
• To quantify the relationship between posterior variance, model capacity, and generalization error.

Main Methods:

• Analyzing mean field inference to demonstrate its information-limiting properties.
• Quantifying model capacity in relation to posterior variance (fixed or learned).
• Connecting information bounds between parameters and data to generalization error.

Main Results:

• Mean field inference limits mutual information between parameters and data via noise, improving generalization.
• A maximum model capacity is identified, linked to generalization error, irrespective of KL-divergence scaling.
• Bounding information between parameters and data acts as an effective regularizer for neural networks.

Conclusions:

• Mean field variational inference provides a principled way to regularize neural networks.
• Understanding and controlling mutual information is key to improving generalization in deep learning models.
• The findings offer theoretical and empirical support for using information-theoretic approaches in model training.