Boosting fine-tuning via Conditional Online Knowledge Transfer.

Area of Science:

• Machine Learning
• Computer Vision
• Deep Learning

Background:

• Fine-tuning enhances network performance with limited labeled data.
• Existing methods use source model knowledge for regularization but can introduce bias.
• Regularization signals are often independent of or loosely coupled with target information.

Purpose of the Study:

• To propose a Conditional Online Knowledge Transfer (COKT) framework.
• To construct robust and target-related regularization signals (RS).
• To improve knowledge transfer during fine-tuning for better network performance.

Main Methods:

• Developed a target-dominant RS branch for online supervision.
• Employed knowledge distillation within the RS branch.
• Integrated target information via sample-wise conditional attention, residual feature fusion, and target task loss.

Main Results:

• COKT significantly outperforms traditional fine-tuning baselines.
• Performance gains are notable for dissimilar target tasks and small datasets.
• The framework demonstrates robustness and adaptability.

Conclusions:

• COKT effectively utilizes target information for more accurate regularization signals.
• The proposed method enhances fine-tuning performance, particularly in challenging scenarios.
• COKT is extensible to cross-model and multi-model fine-tuning settings.