A close look at protein function prediction evaluation protocols.

Area of Science:

• Computational biology
• Bioinformatics
• Protein function prediction

Background:

• The Critical Assessment of Function Annotation challenge (CAFA2) differs from CAFA1 by including predictions for both annotated and unannotated proteins.
• CAFA2 simulates real-world annotation accumulation more realistically.
• This study evaluates the difficulty of these two prediction scenarios.

Purpose of the Study:

• To compare the difficulty of predicting protein function for annotated versus unannotated proteins.
• To assess the reliability of cross-validation for estimating performance in automated function prediction.
• To analyze the performance of various function prediction methods across different CAFA2 tasks.

Main Methods:

• Analysis of protein function prediction method performance on CAFA2 subtasks.
• Comparison of results obtained from cross-validation versus actual task performance.
• Evaluation of structured support vector machine, binary support vector machines, and guilt-by-association methods.

Main Results:

• Predicting novel annotations for previously annotated proteins is more challenging than for uncharacterized proteins.
• Several methods (SVMs, guilt-by-association) show performance discrepancies between cross-validation and actual task performance.
• Cross-validation is insufficient for accurately estimating performance and ranking methods in CAFA2.

Conclusions:

• Findings impact the design of computational experiments for automated function prediction.
• Provides insights for improving future CAFA competitions and method development.
• Highlights the need for more robust evaluation strategies beyond standard cross-validation.