A comprehensive system for evaluation of remote sequence similarity detection.

Area of Science:

• Bioinformatics
• Computational Biology
• Structural Biology

Background:

• Accurate performance evaluation is vital for developing and comparing protein structure prediction methods.
• Existing methods lack a comprehensive evaluation based on a large, unbiased protein set covering all performance aspects.

Purpose of the Study:

• To develop a statistically unbiased benchmark for evaluating sequence-based protein structure prediction methods.
• To create protocols for assessing similarity detection and alignment quality from multiple perspectives.

Main Methods:

• Selected a statistically balanced set of divergent protein domains from SCOP.
• Defined similarity relationships using SCOP data and a Support Vector Machine (SVM) algorithm.
• Developed reference-dependent and reference-independent protocols for evaluating similarity detection and alignment quality.

Main Results:

• The benchmark utilizes ROC-like curves and various approaches for true/false positive definitions.
• Evaluations consider global and local modes for assessing structural match quality.
• Demonstrated how different evaluation aspects reveal distinct properties of evaluated methods.

Conclusions:

• The presented benchmark offers a new tool for statistically unbiased assessment of remote sequence similarity detection methods.
• This tool assists users in selecting appropriate methods and aids developers in creating improved prediction tools.
• The benchmark dataset, alignments, and codes are publicly available for download.