Fold or flop: quality assessment of AlphaFold predictions on whole proteomes

Reliability of AlphaFold2 predictions is mainly assessed using the predicted Local Distance Difference Test (pLDDT). For model organisms, 30-40% of residues fall into the low-confidence pLDDT range. Moreover, pLDDT sometimes fails to flag physically implausible structures. This raises two questions: can more robust reliability indicators be identified, and do unreliable predictions share common structural or biophysical features?

We characterize protein structures through histograms of per-residue neighbor counts, and use the Wasserstein principal component analysis to define the arity map, and lightweight and informative 2D embedding of proteins in a dataset. Using AlphaFold-DB, we show that the arity map reveals three structurally and biophysically distinct populations (well-folded proteins, intrinsically disordered proteins, and physically implausible predictions). We also use our packing based encoding at the residue level to define abstraqt (Arity-Based STRuctural Arrangement Quality assessmenT), a per-residue scoring function complementing the pLDDT, assigning low scores to hallucinated helices and distorted beta strands while correctly scoring native-like predictions.

The code to compute arity maps and rerun the analyses is available within the Structural Bioinformatics Library. See: AlphaFold analysis, and also Documentation, Applications, Installation guide.