Parameter-efficient fine-tuning with layer pruning on medical sequence-to-sequence modeling

The increasing size of language models raises great research interests in parameter-efficient fine-tuning (PEFT) such as LoRA that freezes the main body of a pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note, two public medical dialogue datasets, and is further validated on News summarization dataset XSum and end-to-end text generation dataset E2E. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation quality over ROUGE scores on sequence-to-sequence (Seq2Seq) tasks. The study demonstrated the potential of an integrated PEFT framework, which proves effective in both medical text generation and diverse scenarios.