PepBERT: Lightweight language models for bioactive peptide representation.

Area of Science:

• Bioinformatics
• Computational Biology
• Peptide Science

Background:

• Protein language models (pLMs) are effective for protein and peptide tasks.
• Short peptides (<50 residues) are underrepresented in standard pLM training data (e.g., UniProt Reference Cluster, 2.8%).
• This underrepresentation limits the efficacy of pLMs for peptide-specific applications.

Purpose of the Study:

• To develop a specialized language model for encoding peptide sequences.
• To address the limitations of existing pLMs in handling short peptides.
• To enhance the discovery of bioactive peptides for functional foods.

Main Methods:

• Development of PepBERT, a lightweight peptide language model, with two versions: PepBERT-large (4.9M parameters) and PepBERT-small (1.86M parameters).
• Pretraining PepBERT models from scratch using four custom peptide datasets.
• Evaluation of PepBERT models on nine peptide-related downstream prediction tasks, comparing performance against the benchmark ESM-2 (7.5M parameters).

Main Results:

• PepBERT models achieved superior or comparable performance to ESM-2 on 8 out of 9 evaluated peptide prediction tasks.
• PepBERT provides a compact and efficient solution for generating high-quality peptide representations.
• The models demonstrate effectiveness in encoding peptide sequences, outperforming a larger benchmark model.

Conclusions:

• PepBERT offers a specialized and effective solution for peptide sequence encoding and downstream applications.
• The model can accelerate the discovery of food-derived bioactive peptides with health benefits.
• PepBERT supports the development of sustainable functional foods and the utilization of food processing by-products.