What Do Generative Models Learn About Adaptive Immune Receptor Repertoires? A Benchmark Study

Area of Science:

• Immunoinformatics
• Computational Biology
• Machine Learning

Background:

• Generative models are increasingly used for adaptive immune receptor repertoire (AIRR) sequence analysis.
• Evaluating these models is challenging due to complex data and lack of biological ground truth.
• Existing machine learning metrics are not always suitable for AIRR data.

Purpose of the Study:

• To rigorously evaluate and compare popular generative model families for AIRR sequence data.
• To assess the biological meaningfulness of model outputs beyond surface-level statistics.
• To guide the selection of appropriate models for clinical applications.

Main Methods:

• Applied a suite of evaluation metrics tailored to AIRR sequence data.
• Systematically compared variational autoencoders, LSTMs, antibody language models, selection models, and statistical baselines.
• Focused on learning individual-specific immune receptor repertoires.

Main Results:

• Identified risks of memorization and capabilities for innovation in generated sequences.
• Assessed model sensitivity to hyperparameter tuning.
• Provided a comparative analysis of model performance on AIRR data.

Conclusions:

• Current generative models offer insights into AIRR biology but require careful evaluation.
• Results advance the understanding of model performance in reproducing immune repertoire biology.
• Lays groundwork for principled development and evaluation of AIRR generative models.