GEOMETRY OF LONG-TAILED REPRESENTATION LEARNING: REBALANCING FEATURES FOR SKEWED DISTRIBUTIONS

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Deep learning excels with balanced datasets but struggles with real-world long-tailed distributions.
• Long-tailed data leads to skewed representations, where majority (head) classes dominate the feature space.
• Existing methods empirically correct skewed representations but lack theoretical understanding of the feature skew.

Purpose of the Study:

• To theoretically analyze how long-tailed data distributions affect feature representations.
• To identify the conditions causing tail class feature collapse and overlap.
• To propose a novel method for robust long-tailed representation learning.

Main Methods:

• Theoretical analysis of feature distributions under long-tailed data.
• Derivation of conditions for feature center shrinkage and collapse.
• Development and implementation of the FeatRecon method for feature space reconstruction.

Main Results:

• Demonstrated that tail class centers shrink and can collapse, leading to inseparable features.
• Showed that empirical correction of training data representations is insufficient due to distribution shifts.
• FeatRecon successfully reconstructs feature spaces into symmetrical, linearly separable regions.

Conclusions:

• Long-tailed data fundamentally alters feature distributions, necessitating methods beyond simple empirical correction.
• FeatRecon enhances model robustness by creating separable feature representations for imbalanced datasets.
• The proposed method shows effectiveness across multiple challenging long-tailed benchmarks.