To Transfer or Not to Transfer: Unified Transferability Metric and Analysis | JoVE Visualize

Area of Science:

Machine Learning
Artificial Intelligence
Computer Science

Background:

Transfer learning aims to improve target task performance by leveraging knowledge from a source domain.
Existing transferability estimation methods primarily focus on classification tasks, often overlooking domain/task differences and regression scenarios.
A critical gap exists in determining whether knowledge transfer is beneficial ('transfer or not').

Purpose of the Study:

To propose Wasserstein distance-based joint estimation (WDJE), a unified transferability metric for both classification and regression.
To develop a method for deciding whether to transfer knowledge by comparing target risk with and without transfer.
To estimate the unobservable post-transfer risk using an interpretable upper bound applicable even with limited target labels.

Main Methods:

Developed WDJE, a metric utilizing Wasserstein distance to quantify domain and task differences.
Proposed a nonsymmetric, interpretable upper bound to estimate post-transfer risk, relating it to source model performance and domain/task variations.
Extended the risk bound to unsupervised settings and established generalization bounds.

Main Results:

WDJE demonstrated high accuracy in transferability estimation across 42 scenarios, achieving a perfect consistency index (CI) of 1 in 25 cases and a mean CI of 0.89.
The proposed risk bound showed state-of-the-art performance, achieving high average Pearson correlations (0.99 on CIFAR-100, 0.72 on Office-Home, 0.96 on C-MAPSS) in approximating true post-transfer risk.
The method effectively guides decisions on when to perform knowledge transfer.

Conclusions:

WDJE provides a unified and effective approach for transferability estimation in both classification and regression tasks, addressing domain and task differences.
The proposed risk estimation bound accurately predicts post-transfer performance, even with limited target data, facilitating informed transfer learning decisions.
This research significantly advances the field by providing a robust metric for deciding whether to transfer knowledge, crucial for optimizing machine learning model development.