Robust Differentiable SVD.

Area of Science:

• Numerical analysis
• Computer vision
• Deep learning

Background:

• Eigendecomposition of symmetric matrices is fundamental to computer vision algorithms.
• Numerical instability in eigenvector derivatives, especially with close eigenvalues, hinders deep network integration and convergence.
• Existing methods like Singular Value Decomposition (SVD) and Power Iteration (PI) have limitations in accuracy and scalability.

Purpose of the Study:

• To develop a more numerically stable and accurate method for computing eigenvector gradients in deep learning.
• To address the limitations of existing analytical and iterative approaches for eigendecomposition in large-scale computer vision tasks.
• To improve the convergence and performance of deep networks that rely on eigendecomposition.

Main Methods:

• Utilizing the Taylor expansion of the Singular Value Decomposition (SVD) gradient.
• Deriving eigenvector gradients without relying on iterative processes like Power Iteration (PI).
• Comparing the accuracy and stability of the proposed Taylor expansion method against traditional PI with iterative deflation.

Main Results:

• The Taylor expansion of the SVD gradient is theoretically equivalent to PI gradients but avoids iterative errors.
• The proposed method yields more accurate gradients, mitigating numerical instability issues.
• Demonstrated improved performance in image classification and style transfer tasks due to enhanced gradient accuracy.

Conclusions:

• The Taylor expansion of SVD gradients provides a theoretically sound and practically superior alternative to iterative methods for eigendecomposition in deep learning.
• This approach enhances the reliability and effectiveness of eigendecomposition-based algorithms in computer vision.
• The findings pave the way for more robust and powerful deep learning models leveraging eigendecomposition.