Scaffold-Lab: Critical evaluation and ranking of protein backbone generation methods in a unified framework

Recent advances in de novo protein design have significantly progressed, particularly in protein backbone generation, which remains a challenging yet valuable task. However, there has been a lack in standardized evaluation framework to accurately assess diverse methods, nor is there a comprehensive analysis of their practical applications. In this study, we proposed Scaffold-Lab, a unified framework for systematic evaluation of protein backbone generation methods, encompassing a wide range of metrics including designability, novelty, diversity, efficiency and structural properties. We thoroughly evaluated seven representative methods, providing a detailed analysis of their performance and utilities. Our findings highlight that generating long proteins for unconditional generation and accurately reconstructing motifs are key bottlenecks for most methods. These results underscore both the strengths and limitations of existing protein backbone generation methods, offering new insights for further development and improvement in this field.