Democratizing protein language models with parameter-efficient fine-tuning.

Area of Science:

• Computational Biology
• Proteomics
• Artificial Intelligence

Background:

• Large protein language models (PLMs) have transformed proteomics by learning from vast sequence data.
• Traditional fine-tuning (FT) of PLMs requires significant computational resources, limiting accessibility for many research groups.
• Parameter-efficient fine-tuning (PEFT) methods have addressed similar challenges in natural language processing.

Purpose of the Study:

• To introduce and evaluate PEFT methods for adapting PLMs in proteomics.
• To assess the efficacy of PEFT for protein-protein interaction (PPI) prediction and homooligomer symmetry prediction.
• To provide a resource for democratizing PLM adaptation in proteomics.

Main Methods:

• Leveraged the LoRA (Low-Rank Adaptation) method for PEFT of PLMs.
• Trained models for predicting protein-protein interactions (PPIs) and homooligomer quaternary structure symmetry.
• Conducted a comprehensive evaluation of the hyperparameter space for PEFT in proteomics.

Main Results:

• PEFT approaches demonstrated competitive performance compared to traditional FT for both tasks.
• PEFT methods required substantially fewer parameters and reduced memory footprint.
• Training only the classification head for PPI prediction was highly effective, using significantly fewer parameters than full FT.
• PEFT methods outperformed state-of-the-art PPI prediction techniques with reduced computational cost.
• PEFT was found to be robust to hyperparameter variations.

Conclusions:

• PEFT offers a computationally efficient and effective alternative to traditional FT for PLMs in proteomics.
• These methods democratize access to advanced PLM adaptation for researchers with limited computational resources.
• Best practices for PEFT in proteomics may differ from those in natural language processing, requiring further investigation.