Locality-aware pooling enhances protein language model performance across varied applications.

Area of Science:

• Computational biology
• Bioinformatics
• Machine learning in biology

Background:

• Protein language models (PLMs) analyze protein sequences using transformer architectures.
• Current PLMs generate contextualized amino acid representations.
• Per-residue embeddings are pooled into fixed-size vectors for downstream tasks, but common methods like Cls-Pooling and Avg-Pooling miss local substructures and long-range interactions.

Purpose of the Study:

• To introduce attention pooling for capturing local substructures and long-range interactions in proteins.
• To develop an efficient pooling strategy, Bag-of-mer pooling (BoM-Pooling), by combining windowed average pooling with attention pooling.
• To enhance the effectiveness of protein sequence modeling using biologically inspired pooling techniques.

Main Methods:

• Proposed attention pooling to capture protein-specific features.
• Introduced Bag-of-mer pooling (BoM-Pooling), a hierarchical pooling technique.
• Combined windowed average pooling with attention pooling for computational feasibility.

Main Results:

• Attention pooling and BoM-Pooling outperform traditional pooling strategies.
• Demonstrated superior performance on predicting protein activities, detecting remote homologs, and predicting signaling protein interactions.
• Highlighted the advantages of biologically inspired pooling in protein sequence modeling.

Conclusions:

• Attention pooling effectively captures essential protein features.
• BoM-Pooling offers an efficient and effective pooling strategy for PLMs.
• Biologically inspired pooling techniques represent a significant advancement for language models in biological applications.