DeCAF: Decentralized consensus-and-factorization for low-rank adaptation of foundation models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Low-Rank Adaptation (LoRA) is a key fine-tuning method for large models, enabling efficient training on edge devices.
• Decentralized LoRA (DLoRA) applications are underexplored due to issues with gradient smoothness and model consensus interference.
• Existing decentralized training methods face challenges with convergence and data distribution variations.

Purpose of the Study:

• To improve the convergence rate of decentralized LoRA (DLoRA) to match that of decentralized Stochastic Gradient Descent (SGD).
• To introduce a novel algorithm, DeCAF, that resolves consensus interference in DLoRA using truncated singular value decomposition (TSVD).
• To provide theoretical guarantees and empirical validation for the proposed algorithms in decentralized learning environments.

Main Methods:

• Ensuring gradient smoothness to enhance DLoRA convergence rates.
• Integrating DLoRA with TSVD-based matrix factorization in the DeCAF algorithm to mitigate consensus interference.
• Conducting theoretical analysis to bound TSVD approximation error and demonstrate vanishing consensus differences.

Main Results:

• Achieved improved convergence rates for DLoRA, matching decentralized SGD.
• Demonstrated that DeCAF effectively resolves consensus interference, leading to comparable convergence rates.
• Experimental results show superior performance of the proposed algorithms over local training and federated learning on both IID and Non-IID data.

Conclusions:

• The developed algorithms offer significant improvements for decentralized fine-tuning of large models.
• DeCAF provides a robust solution for consensus interference, enhancing DLoRA's effectiveness in distributed settings.
• The findings support the broader applicability of efficient fine-tuning techniques in decentralized machine learning.