Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

Area of Science:

• Numerical optimization
• Machine learning theory

Background:

• Optimization over the Stiefel manifold is crucial for machine learning.
• Current methods involve costly orthonormalization or sensitive parameter tuning.

Purpose of the Study:

• To develop a retraction-free and penalty parameter-free algorithm for Stiefel manifold optimization.
• To introduce Manifold-LoRA for geometry-accelerated adaptation of large language models.

Main Methods:

• A novel landing technique directly on the manifold, leveraging quadratic penalty properties and proximal smoothness.
• Reformulation of low-rank adaptation (LoRA) as a manifold optimization problem.

Main Results:

• Global convergence guarantees with state-of-the-art iteration complexities.
• Demonstrated efficiency and strong downstream performance of Manifold-LoRA in numerical experiments.

Conclusions:

• The proposed algorithm offers an efficient and robust alternative for Stiefel manifold optimization.
• Manifold-LoRA accelerates large language model adaptation through geometric insights.