Unsupervised Knowledge Transfer Using Similarity Embeddings.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) are increasingly complex and computationally intensive.
• Transferring knowledge from large DNNs to smaller ones is crucial for efficiency.
• Existing knowledge transfer methods often rely on soft-targets or direct weight usage.

Purpose of the Study:

• To propose a novel unsupervised knowledge transfer method for DNNs.
• To enable knowledge transfer between any two layers of neural networks, irrespective of neuron count.
• To avoid lossy dimensionality reduction and minimize reliance on the source model's architecture.

Main Methods:

• The proposed method utilizes similarity-induced embeddings for knowledge transfer.
• It requires only the activations from a specific layer of the source model.
• The technique facilitates transfer between any two layers, regardless of their size.

Main Results:

• The method was evaluated on six diverse image datasets.
• Extensive experiments demonstrated successful knowledge transfer.
• The approach is effective with various data types, including synthetic and cross-domain data.

Conclusions:

• The proposed similarity-induced embedding method offers an effective approach for unsupervised knowledge transfer.
• This technique allows flexible knowledge transfer between arbitrary network layers.
• The method shows robustness across different datasets and data types, highlighting its practical applicability.