Generalized Kullback-Leibler Divergence Loss

Area of Science:

• Machine Learning
• Computer Vision
• Optimization

Background:

• Kullback-Leibler (KL) Divergence loss is a fundamental metric in machine learning.
• Existing KL loss formulations present limitations in specific applications like knowledge distillation and adversarial training.
• The decoupled structure of Decoupled KL (DKL) Divergence loss offers potential for improvement.

Purpose of the Study:

• To mathematically prove the equivalence between KL Divergence loss and Decoupled KL (DKL) Divergence loss.
• To enhance KL/DKL loss by addressing optimization challenges and sample bias.
• To introduce a novel Generalized KL (GKL) Divergence loss.

Main Methods:

• Mathematical proof of KL and DKL loss equivalence.
• Modification of KL loss to break asymmetric optimization and incorporate smoother weight functions.
• Integration of class-wise global information into KL/DKL loss.
• Empirical evaluation on CIFAR-10/100, ImageNet, and vision-language datasets.

Main Results:

• Demonstrated equivalence between KL Divergence loss and DKL loss (weighted Mean Square Error + Cross-Entropy with soft labels).
• Achieved state-of-the-art adversarial robustness on the RobustBench leaderboard.
• Obtained competitive knowledge distillation performance on various models and datasets.
• The proposed Generalized KL (GKL) Divergence loss shows significant practical merits.

Conclusions:

• The Generalized KL (GKL) Divergence loss offers substantial improvements over standard KL and DKL losses.
• GKL loss effectively enhances adversarial robustness and knowledge distillation tasks.
• The findings provide a more robust and versatile loss function for deep learning applications.