Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization

Background:

• Deep neural networks (DNNs) exhibit complex training dynamics.
• Understanding DNNs trained for extended periods is crucial for advancing AI.
• Existing models struggle to fully capture DNN behavior in imbalanced datasets.

Purpose of the Study:

• Introduce the Layer-Peeled Model, an optimization program for analyzing DNNs.
• Explain phenomena like neural collapse and discover new ones like Minority Collapse.
• Provide insights into mitigating performance limitations in minority classes.

Main Methods:

• Developed the Layer-Peeled Model by isolating and constraining the top layer of a neural network.
• Analyzed the model's solutions on class-balanced and imbalanced datasets.
• Utilized mathematical proofs and computational experiments to validate findings.

Main Results:

• Proved that solutions for balanced datasets form simplex equiangular tight frames, explaining neural collapse.
• Discovered and defined Minority Collapse, a phenomenon limiting performance on minority classes in imbalanced datasets.
• The Layer-Peeled Model predicted Minority Collapse before experimental confirmation.

Conclusions:

• The Layer-Peeled Model offers a tractable approach to understanding deep learning training.
• It provides a theoretical basis for neural collapse and reveals critical insights into Minority Collapse.
• The model serves as a tool for predicting and potentially mitigating performance issues in DNNs, especially with imbalanced data.