BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models

Area of Science:

• Natural Language Processing
• Machine Learning
• Artificial Intelligence

Background:

• Parameter Efficient Fine-Tuning (PEFT) is crucial for adapting large language models.
• Manual PEFT design often leads to suboptimal results.
• Existing automatic PEFT methods struggle with search space complexity and efficiency.

Purpose of the Study:

• To develop a more efficient and effective automatic PEFT strategy.
• To address challenges in search space entanglement and parameter budget integration.
• To enhance the performance of PEFT for downstream tasks.

Main Methods:

• Introduced Budget-guided Iterative search strategy for automatic PEFT (BIPEFT).
• Employed an iterative search to disentangle binary module and rank dimension search spaces.
• Designed early selection strategies guided by parameter budgets.

Main Results:

• BIPEFT significantly enhances search efficiency in automatic PEFT.
• Early selection strategies accelerate learning by removing unimportant modules.
• Demonstrated superior performance of BIPEFT on public benchmarks with low parameter budgets.

Conclusions:

• BIPEFT offers an efficient and effective solution for automatic PEFT.
• The budget-guided iterative approach overcomes limitations of previous methods.
• Achieves high performance in downstream tasks using minimal parameters.