Differentiable Optimization of Generalized Nondecomposable Functions using Linear Programs.

Area of Science:

• Machine Learning
• Deep Learning
• Optimization

Background:

• Training deep neural networks with non-decomposable performance measures (e.g., AUC, F-measure) is challenging.
• Existing methods often rely on surrogate lower bounds or specialized optimization schemes.

Purpose of the Study:

• To propose a feasible framework for directly training deep neural networks with non-decomposable performance measures.
• To develop an efficient and generalizable optimization strategy applicable to various task-specific metrics.

Main Methods:

• The framework involves solving linear programs (LPs) during training, where network representations define the LP constraints.
• A generalized Newton method, adapted from 1-norm SVMs, is employed to solve LPs efficiently on GPUs.
• Sufficient dimension reduction is utilized to handle large constraint matrices in settings like large mini-batches.

Main Results:

• The proposed method achieves superior computational behavior and performance improvements compared to existing alternatives.
• The approach is applicable without specific adjustments or relaxations for different use cases.
• Efficient GPU implementation and effective handling of large-scale problems are demonstrated.

Conclusions:

• The developed framework offers a direct and effective way to train deep neural networks with non-decomposable performance measures.
• The generalized Newton method and dimension reduction techniques provide a robust and efficient optimization solution.
• This work advances the training of deep learning models for tasks requiring complex performance metrics.