Evaluating the effectiveness of sequence analysis algorithms using measures of relevant information

Given vast quantities of molecular sequence data, and numerous different algorithms designed to discover, diagnose or model biologically interesting features in sequences, how is it possible to make objective evaluations of the diagnostic effectiveness of these algorithms and robust assessments of their relative strengths and limitations? An approach to this relatively neglected question is developed here, which is based on information measures of the diagnostic efficiency of different methods. From output lists of a procedure such as a database search, "relevance weights" are assigned that encode, for each sequence listed, the level of associated scientific evidence implicating that sequence as an example of a feature of interest. Relevance weights may be derived, following systematic protocols, from expert human judgement or, in principle, by automated information retrieval from electronic resources. Practical applications of this approach to algorithm assessment and development and parameter choice are demonstrated with examples of automated sequence motif modeling for the DNA-binding helix-turn-helix motif and the guanine exchange factor protein domain. The combined use of relevance weights and information measures appears to offer promising advantages over ROC analysis and may be generally applicable to diagnostic evaluation.