Benchmarking Generative AI Protein Models Reveals Differences Between Structural and Sequence-based Approaches.

Area of Science:

• Biochemistry
• Artificial Intelligence
• Computational Biology

Background:

• Generative artificial intelligence (AI) models are advancing de novo protein design.
• Evaluating these models is crucial for their application in bio-medical engineering.

Purpose of the Study:

• To systematically benchmark 13 state-of-the-art generative protein models.
• To assess model performance in generating feasible, diverse, and novel protein monomers.
• To compare structural diffusion models and protein language models for protein design.

Main Methods:

• Comparative analysis of 13 generative protein models.
• Evaluation of protein monomer feasibility, diversity, and novelty.
• Conditional generation of proteins based on Tobacco Etch Virus (TEV) protease.

Main Results:

• Structural diffusion models yield high-confidence, plausible designs but lack diversity and show sequence biases.
• Protein language models generate diverse, novel designs with lower structural confidence.
• Generative models successfully produced functional enzymes, though with reduced activity compared to wildtype TEV.

Conclusions:

• Generative protein models exhibit complementary strengths, with diffusion models excelling in structural accuracy and language models in design diversity.
• A systematic benchmarking framework is established for evaluating and selecting generative protein models.
• This research facilitates informed application of AI tools for bio-medical engineering and protein design.