Tangent-space gradient optimization of tensor network for machine learning.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Probabilistic Modeling

Background:

• Deep learning models often encounter gradient vanishing and exploding issues, hindering training in deep computational graphs.
• Existing optimization methods may require adaptive learning rate adjustments, adding complexity.

Purpose of the Study:

• To introduce a novel optimization method, tangent-space gradient optimization (TSGO), to mitigate gradient instability in probabilistic models.
• To ensure stable gradient propagation regardless of computational graph depth.

Main Methods:

• TSGO guarantees orthogonality between variational parameters and gradients by rotating the parameter vector towards the gradient direction.
• The method is demonstrated within tensor network (TN) machine learning, representing probability distributions in Hilbert space.
• The learning rate in TSGO is intrinsically defined by the rotation angle θ as η=tanθ.

Main Results:

• TSGO effectively restricts gradients within the tangent space of the normalization hypersphere (〈ψ|ψ〉=1).
• Numerical experiments show TSGO achieves superior convergence compared to the standard Adam optimizer.
• The proposed method eliminates the need for external adaptive learning rate controls.

Conclusions:

• Tangent-space gradient optimization (TSGO) offers a robust solution to gradient instability in deep probabilistic models.
• TSGO provides a principled and effective alternative to conventional gradient-based optimization techniques like Adam.
• The inherent learning rate determination simplifies implementation and improves training stability.