Efficient gradient computation for optimization of hyperparameters.

Area of Science:

• Machine Learning
• Optimization
• Medical Imaging

Background:

• Learning hyperparameters in convex objective functions is crucial for supervised learning.
• Neural networks can model complex relationships between input data and hyperparameters.
• Efficient gradient computation is needed for end-to-end training in optimization-based frameworks.

Purpose of the Study:

• Develop general formulas for gradient backpropagation in convex problems, specifically proximal mappings.
• Illustrate the application of these formulas to 1D quadratic smoothing using dynamic programming.
• Enable efficient end-to-end training of hyperparameters in machine learning models.

Main Methods:

• Derived general formulas for gradient backpropagation of proximal mappings.
• Developed a dynamic programming (DP) algorithm for exact gradient computation of hyperparameters in 1D quadratic smoothing.
• Compared custom gradient computation with automatic differentiation in deep learning libraries.

Main Results:

• The proposed DP algorithm for gradient computation achieved 55%-65% computation time savings.
• Custom gradients are more efficient than automatic differentiation for specific convex problems.
• The developed formulas and strategy are applicable to other smoothing problems like TV and Huber smoothing.

Conclusions:

• The general formulas for gradient backpropagation significantly improve computational efficiency.
• This work provides a practical method for training hyperparameters in optimization-based machine learning.
• The approach is extensible to various convex smoothing problems in medical imaging and beyond.