Automated large scale evaluation of protein structure predictions

Evaluation and assessment are critical issues in CASP experiments. Automated procedures are necessary to compare a large number of predictions with the target folds. The evaluation has to reveal the maximum extent of similarity between predictions and targets, it should be applicable across prediction categories, and it should be transparent and accessible to a wide community. Here we present an automated evaluation scheme which is an attempt to meet these requirements. In the implementation and execution of this scheme we had to solve or circumvent problems of convergence, where algorithms fail to find optimum solutions, problems of ambiguity where no unique optimum solution exists, and problems in ranking and interpretation. Key features of this implementation are (1) the root mean square deviation of structure superimposition is kept close to a constant value throughout the evaluation and (2) all structural matches found between two folds are taken into account. We discuss these points in detail and describe the numerical criteria used in the CASP3 evaluation.