Parameter-efficient fine-tuning with layer pruning on medical sequence-to-sequence modeling.

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Machine Learning

Background:

• Large language models (LLMs) require efficient fine-tuning methods.
• Parameter-efficient fine-tuning (PEFT) techniques like LoRA freeze most parameters and add trainable ones for downstream tasks.
• Further optimization is needed to reduce computational costs.

Purpose of the Study:

• To propose and validate an integrated framework combining LoRA and structured layer pruning for enhanced PEFT.
• To assess the framework's efficiency in terms of memory usage and training speed.
• To evaluate the impact on generation quality across various NLP tasks.

Main Methods:

• Integrated LoRA with structured layer pruning.
• Tuned only 0.6% of model parameters.
• Pruned over 30% of Transformer layers.
• Validated on medical report summarization, medical dialogue, news summarization, and text generation datasets.

Main Results:

• Reduced GPU memory usage by 50%.
• Increased training speed by 100%.
• Preserved over 92% of generation quality based on ROUGE scores for Seq2Seq tasks.
• Demonstrated effectiveness on diverse datasets, including medical and general text generation.

Conclusions:

• The integrated PEFT framework offers significant efficiency gains for LLMs.
• This approach is effective for both specialized (medical) and general NLP tasks.
• The method balances computational efficiency with high performance in text generation.