S²ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning

Area of Science:

• Biotechnology
• Computational Biology
• Immunology

Background:

• Antibodies are crucial for health and disease treatment, with biomedical language models showing promise.
• Current models lack explicit consideration of antibody structural information, limiting their predictive power.
• Both 1D sequence and 3D structure offer complementary insights into antibody function.

Purpose of the Study:

• To propose a unified antibody language model, S2ALM, that integrates both sequential and structural information.
• To develop a hierarchical pre-training paradigm with customized objectives for comprehensive antibody representations.
• To demonstrate S2ALM's utility in various downstream tasks, including binding affinity prediction and therapeutic antibody design.

Main Methods:

• Developed the Sequence-Structure multi-level pre-trained Antibody Language Model (S2ALM).
• Employed a hierarchical pre-training paradigm with two customized multi-level training objectives.
• Pre-trained S2ALM on a large dataset of 75 million sequences and 11.7 million structures.

Main Results:

• S2ALM's representation space reveals functional binding mechanisms, evolutionary properties, and structural interaction patterns.
• Achieved state-of-the-art performance in predicting antigen-antibody binding affinities and B cell maturation stages.
• Demonstrated success in identifying crucial antibody binding sites and designing novel coronavirus-binding antibodies.

Conclusions:

• S2ALM effectively models comprehensive and generalized antibody representations by integrating sequence and structure.
• The model shows significant potential for advancing therapeutic antibody development and addressing unmet needs.
• S2ALM outperforms existing baselines across diverse antibody understanding and generation tasks.