Attentive Learning Facilitates Generalization of Neural Networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural networks (NNs) often struggle with generalization, especially when exposed to out-of-distribution (OoD) data during training.
• Understanding how NNs learn from noisy or irrelevant data is crucial for improving their reliability and performance.

Purpose of the Study:

• To investigate the impact of OoD examples on neural network generalization.
• To introduce and quantify a new metric, dataset-distraction stability, to measure a network's resilience to irrelevant training data.
• To establish a theoretical link between attentive learning, dataset-distraction stability, and generalization bounds.

Main Methods:

• Examined neural network behavior when trained on datasets with and without OoD examples.
• Proposed and measured dataset-distraction stability across various NN architectures (VGG, ResNet, WideResNet, ViT) and optimizers.
• Utilized intrinsic dimensions (IDs) to decompose the learning process on complex distributions into simpler ones.

Main Results:

• A negative correlation was observed between dataset-distraction stability and generalizability in extensive CIFAR-10/100 experiments.
• Networks with higher distraction stability exhibited better generalization.
• A tighter generalization bound was derived by decomposing the learning process based on distribution simplicity.

Conclusions:

• Attentive learning, characterized by low influence from OoD examples, is key to achieving strong generalization in deep learning.
• Dataset-distraction stability provides a quantifiable measure to understand and potentially improve generalization.
• The findings pave the way for designing novel algorithms that promote attentive learning and enhance model robustness.