Aligning Logits Generatively for Principled Black-Box Knowledge Distillation in the Wild

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Black-Box Knowledge Distillation (B2KD) is crucial for cloud-to-edge model compression, protecting data privacy and model copyrights.
• B2KD utilizes only teacher model API query results for distilling lightweight student models for edge devices.
• Challenges include limited internet exchange and edge-cloud data distribution disparities.

Purpose of the Study:

• To address B2KD challenges by proposing a novel optimization direction from logits to cell boundary.
• To introduce Mapping-Emulation KD (MEKD) for robust edge model prediction and anti-interference capabilities.
• To enhance student model performance on edge devices across diverse, unknown data distributions.

Main Methods:

• A three-stage workflow: deprivatization (emulating inverse teacher mapping), distillation (aligning low-dimensional logits via high-dimensional image point distances), and adaptation (graph propagation-based test-time adaptation).
• MEKD emulates the teacher's inverse function using a generator for deprivatization.
• Low-dimensional logit alignment is achieved by minimizing distances between high-dimensional image representations.

Main Results:

• MEKD demonstrates strong performance in edge model distillation and adaptation across various teacher-student architectures.
• State-of-the-art results were achieved on multiple image recognition benchmarks using diverse Deep Neural Network models.
• The method shows practical value, particularly in remote sensing image recognition applications.

Conclusions:

• MEKD effectively enhances the robustness and anti-interference capabilities of student models for edge deployment.
• The proposed approach overcomes limitations of direct logit alignment in B2KD.
• MEKD offers a practical and effective solution for privacy-preserving model compression in real-world edge AI scenarios.