Protein FID: improved evaluation of protein structure generative models

Area of Science:

• Computational biology
• Structural bioinformatics
• Machine learning in protein science

Background:

• Generative models for protein structure are rapidly advancing.
• Existing evaluation metrics do not fully assess the distributional coverage of model outputs.
• A need exists for metrics that capture semantic similarity in protein structure generation.

Purpose of the Study:

• To propose and validate a Frechet Inception Distance (FID) metric for evaluating protein structure generative models.
• To assess the distributional similarity of generated protein structures within a meaningful latent space.
• To benchmark current protein structure generative models against the Protein Data Bank distribution.

Main Methods:

• Developed a protein-specific Frechet Inception Distance (FID) metric.
• Utilized a semantically meaningful latent space for comparing protein structure distributions.
• Correlated FID with optimal transport distances and hierarchical protein classifications (CATH).

Main Results:

• The proposed protein FID metric demonstrates desirable behavior under structural perturbations.
• FID successfully recapitulates known similarities between protein structures, aligning with FoldSeek clusters and the CATH hierarchy.
• Current protein structure generative models, when evaluated with FID, show limitations in capturing the full distribution of Protein Data Bank (PDB) proteins.

Conclusions:

• Protein FID offers a valuable supplement to existing metrics for evaluating protein structure generative models.
• The metric provides insights into the distributional sampling capabilities of these models.
• Further development and application of protein FID can drive progress in generative protein design.