Triplet Decomposition and Extensions: A General Framework for Parameter-Efficient Fine-Tuning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Parameter-Efficient Fine-Tuning (PEFT) methods are crucial for adapting large pre-trained models to specific tasks with reduced computational cost.
• Existing PEFT methods often utilize low-rank decomposition, assuming linear patterns in weight updates, which may not capture complex model adaptations.
• The statistical properties of weight updates during fine-tuning can be more intricate than simple low-rank structures.

Purpose of the Study:

• To investigate and unify existing Parameter-Efficient Fine-Tuning (PEFT) approaches under a novel Triplet Matrix Decomposition framework.
• To explore the expressivity of frequency-domain decomposition for weight updates compared to traditional low-rank methods.
• To introduce a new PEFT method, Learnable Orthogonal Adaptation (LoTA), that leverages insights from frequency-domain analysis and orthogonal transformations.

Main Methods:

• Unified low-rank and frequency-domain decomposition methods using a Triplet Matrix Decomposition framework for comparative analysis.
• Proposed Learnable Orthogonal Adaptation (LoTA), a novel PEFT method employing cascaded Householder transformations for orthogonal matrix construction.
• Utilized finite-difference gradient approximation to enable end-to-end backpropagation for discrete basis selection optimization.

Main Results:

• Frequency-domain methods demonstrate superior expressivity over low-rank approaches when optimal frequency components are selected, attributed to orthogonal transformations and flexible basis combinations.
• Learnable Orthogonal Adaptation (LoTA) achieved superior parameter efficiency and performance across various tasks, including natural language understanding, mathematical reasoning, commonsense reasoning, computer vision, and visual instruction tuning.
• LoTA effectively learns task-specific transformations and adaptive basis combinations, outperforming existing PEFT techniques.

Conclusions:

• Frequency-domain decomposition offers a more expressive paradigm for Parameter-Efficient Fine-Tuning (PEFT) compared to low-rank methods.
• Learnable Orthogonal Adaptation (LoTA) presents a novel and effective PEFT strategy, achieving state-of-the-art performance and parameter efficiency.
• The proposed method successfully addresses challenges in optimizing basis selection for complex weight updates in large model adaptation.