Inducing Neural Collapse via Anticlasses and One-Cold Cross-Entropy Loss

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Standard softmax cross-entropy (CE) loss overlooks relationships between nontarget classes, leaving optimization information unexploited.
• This limitation hinders model performance by failing to leverage complementary class data effectively.

Purpose of the Study:

• To propose a novel loss function, One-Cold CE (OCCE) loss, to address the limitations of standard CE loss.
• To structure the activations of complementary classes for improved feature representation.
• To enhance model generalization and performance across various machine learning tasks.

Main Methods:

• Defined an 'anticlass' for each target class, encompassing all non-target instances including complementary classes and out-of-distribution samples.
• Implemented a uniform one-cold encoded distribution target for each anticlass.
• Encouraged models to equally distribute activations across all nontarget classes during optimization.

Main Results:

• Promoted a symmetric geometric structure of classes in the feature space.
• Increased the degree of neural collapse (NC) during training.
• Addressed the independence deficit problem in neural networks, leading to improved generalization.

Conclusions:

• The proposed OCCE loss consistently enhances performance in classification, open-set recognition, and out-of-distribution detection tasks.
• OCCE loss effectively exploits information from complementary classes, leading to more robust and generalizable models.
• This novel approach offers a significant improvement over standard CE loss for supervised classification and related tasks.