Information Losses in Neural Classifiers From Sampling.

Area of Science:

• Machine Learning
• Information Theory
• Neural Networks

Background:

• Finite datasets in neural network training lead to information losses.
• Existing bounds on these losses have limitations, particularly concerning input compression and model complexity.

Purpose of the Study:

• To derive novel bounds for information losses in neural classifiers trained on finite datasets.
• To provide a theoretical explanation for recent experimental findings on information compression in neural networks.

Main Methods:

• Establishing a relationship between information loss and the expected total variation of the neural model.
• Bounding the expected total variation as a function of dataset size in a general setting.
• Comparing derived bounds with existing bounds and experimental results.

Main Results:

• Information loss bounds are derived as the product of expected total variation and feature space information in hidden representations.
• New bounds are less sensitive to input compression and generally smaller than existing ones.
• The derived bounds successfully explain recent experimental findings on information compression in neural networks.

Conclusions:

• The novel bounds offer a more accurate and sensitive measure of information loss in neural network training.
• These findings advance the understanding of information compression mechanisms within neural networks.
• The theoretical bounds align well with empirical observations, validating their utility.