Compressing the collective knowledge of ESM into a single protein language model

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Protein language models (PLMs) are emerging for variant-effect prediction (VEP).
• Current high-performing VEP methods often integrate diverse data types (homology, structure, population genetics) with PLMs, increasing complexity.
• Pure sequence-based PLMs, like Evolutionary Scale Modeling (ESM), offer simplicity but may have limitations.

Purpose of the Study:

• To challenge the notion that sequence-only PLMs are inherently limited for VEP.
• To present an efficient co-distillation technique for adapting sequence-only PLMs for high-accuracy VEP.
• To demonstrate that this method can achieve state-of-the-art performance without external data.

Main Methods:

• Developed an efficient co-distillation approach to adapt PLMs for VEP.
• Enabled individual PLMs to self-improve by distilling confident predictions from multiple models within the same family.
• Focused on utilizing evolutionary signals captured during pretraining, avoiding additional data sources.

Main Results:

• Co-distillation of ESM models achieved state-of-the-art performance across multiple VEP benchmarks.
• The enhanced PLMs accurately quantified variant effects on continuous clinical phenotypes using biobank data.
• Demonstrated that sequence-only PLMs, when co-distilled, can match or exceed the performance of methods using complex, multi-modal data.

Conclusions:

• Co-distillation is an effective strategy to enhance sequence-only PLMs for VEP.
• This method simplifies VEP by removing the need for complex, external data integration.
• The improved VEP accuracy has significant implications for interpreting genetic variation in clinical contexts and biobank-scale studies.