A curriculum learning approach to training antibody language models.

Area of Science:

• * Computational biology
• * Bioinformatics
• * Machine learning in immunology

Background:

• * Antibody language models (AbLMs) are increasingly pre-trained using both unpaired and paired antibody sequences.
• * Combining these datasets offers potential benefits but faces challenges due to data imbalance and lack of systematic evaluation for data processing and training strategies.
• * Existing methods struggle to fully leverage the scale of unpaired data alongside the specificity of paired data.

Purpose of the Study:

• * To develop and evaluate a systematic method for pre-training AbLMs using a mixture of unpaired and paired antibody sequences.
• * To address the data imbalance challenge inherent in combining large-scale unpaired datasets with smaller, high-quality paired datasets.
• * To optimize training strategies that maximize the utility of both data types for improved AbLM performance.

Main Methods:

• * Introduction of a curriculum learning approach for AbLMs, enabling a phased training process from unpaired to paired sequences.
• * Optimization of data processing and training methodologies tailored for mixed-data AbLM pre-training.
• * Development and evaluation of a 650M-parameter model named CurrAb.

Main Results:

• * The developed curriculum learning method effectively integrates unpaired and paired antibody sequences during AbLM pre-training.
• * The CurrAb model, utilizing this method, significantly outperforms existing mixed AbLMs on downstream classification tasks.
• * Optimized training strategies successfully mitigate the challenges posed by data imbalance.

Conclusions:

• * Curriculum learning presents a viable and effective strategy for pre-training AbLMs on mixed sequence datasets.
• * The CurrAb model establishes a new benchmark for performance in antibody sequence-based classification tasks.
• * This work provides a foundational framework for future development of AbLMs leveraging diverse antibody sequence data.