Fine-tuning protein language models boosts predictions across diverse tasks

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning

Background:

• Protein language models (PLMs) generate embeddings that achieve state-of-the-art performance in protein prediction.
• Unlike natural language processing, PLM embeddings are often not fine-tuned for downstream protein prediction tasks.

Purpose of the Study:

• To investigate the impact of fine-tuning on state-of-the-art protein language models (ESM2, ProtT5, Ankh).
• To compare full model fine-tuning with parameter-efficient fine-tuning (PEFT) across eight diverse protein prediction tasks.

Main Methods:

• Evaluated three leading protein language models: ESM2, ProtT5, and Ankh.
• Applied task-specific supervised fine-tuning to each model.
• Compared full parameter fine-tuning with parameter-efficient fine-tuning strategies.
• Assessed performance across eight distinct protein prediction benchmarks.

Main Results:

• Task-specific fine-tuning consistently improved downstream prediction performance.
• Parameter-efficient fine-tuning achieved comparable improvements with significantly reduced computational resources.
• PEFT accelerated training by up to 4.5-fold compared to full model fine-tuning.

Conclusions:

• Fine-tuning protein language models is highly recommended for enhancing prediction accuracy, especially for small datasets.
• Parameter-efficient fine-tuning presents a computationally efficient and effective approach for adapting PLMs.
• The study provides accessible notebooks for fine-tuning various PLMs for per-protein and per-residue prediction tasks.