LIFT+: Lightweight Fine-Tuning for Long-Tail Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Foundation Models

Background:

• Fine-tuning is crucial for long-tail learning with foundation models.
• Current fine-tuning strategies' impact on long-tail performance is not well understood.
• Existing methods may misuse fine-tuning, impacting efficiency and accuracy.

Purpose of the Study:

• To investigate the effects of different fine-tuning strategies on long-tail learning.
• To identify the causes of performance deterioration in tail classes.
• To propose an improved lightweight fine-tuning framework.

Main Methods:

• Theoretical and empirical validation of fine-tuning impacts.
• Analysis of class conditional distributions under heavy fine-tuning.
• Development and implementation of the LIFT+ framework.
• Incorporation of semantic-aware initialization, minimalist data augmentation, and test-time ensembling.

Main Results:

• Heavy fine-tuning degrades performance on tail classes due to inconsistent distributions.
• Lightweight fine-tuning proves more effective for long-tail learning.
• LIFT+ framework optimizes class conditions for enhanced adaptation and generalization.
• Significant reductions in training epochs (90%) and learned parameters (<1%) achieved.

Conclusions:

• Lightweight fine-tuning is superior for long-tail learning with foundation models.
• LIFT+ offers an efficient and accurate solution, surpassing state-of-the-art methods.
• The framework enhances model adaptation, generalization, convergence, and compactness.