A measure of progress in fold recognition?

We present a retrospective analysis of CASP3 threading predictions, applying evaluation and assessment criteria used at CASP2. Our purpose is twofold. First, we wish to ask whether measures of model accuracy are comparable between CASP3 and CASP2, even though they have been calculated differently. We find that these quantities are effectively the same, and that either may be used to compare model accuracy. Secondly, we wish to assess progress in fold recognition by comparing the numbers of CASP2 and CASP3 models that cross specific accuracy thresholds. We find that the number of accurate models at CASP3 drops sharply as the targets become more difficult, with less extensive similarity to known structures, exactly the pattern seen at CASP2. CASP3 teams do not seem to have predicted accurate models for targets of greater difficulty, and for a given difficulty range the best CASP3 models seem no more accurate than the best models at CASP2. At CASP3, however, we find greater numbers of accurate models for medium-difficulty targets, with extensive similarity to a known structure but no shared sequence motifs. Threading methods would appear to have become more reliable for modeling based on remote evolutionary relationships.