Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in CASP12

Area of Science:

• Computational Biology
• Structural Bioinformatics
• Protein Structure Prediction

Background:

• The Critical Assessment of Techniques for Structure Prediction (CASP) offers biennial assessments of protein structure prediction methods.
• Developers of automated computational modeling methods require more frequent evaluations using larger datasets.

Purpose of the Study:

• To introduce the Continuous Automated Model EvaluatiOn (CAMEO) platform for frequent, automated benchmarking of protein structure prediction methods.
• To complement CASP by providing weekly assessments and facilitating method development.

Main Methods:

• CAMEO conducts fully automated blind prediction assessments using weekly pre-release protein sequences from the Protein Data Bank.
• Benchmarking results are published weekly, based on models collected within a 4-day window, assessing approximately 100 targets over 5 weeks.
• Consistent data generation allows developers to benchmark and cross-validate method performance.

Main Results:

• CAMEO provides weekly benchmarking data, enabling consistent performance evaluation for participating methods.
• The platform facilitates server development and shorter release cycles through weekly performance statistics and warnings.
• Many CASP participants utilize CAMEO to prepare for upcoming community experiments.

Conclusions:

• CAMEO offers a valuable, continuous assessment framework for protein structure prediction methods, complementing CASP.
• The platform supports method development, validation, and faster iteration cycles in computational modeling.
• CAMEO's scoring schemes encourage advancements in areas like quaternary structure and complex modeling.