The Null-Test for peptide identification algorithm in Shotgun proteomics

The present research proposed general evaluation strategy named Null-Test for peptide identification algorithm in Shotgun proteomics. The Null-Test method based on random matching can be utilized to check whether the algorithm has a tendency to make a mistake or has potential bugs, faultiness, errors etc., and to validate the reliability of the identification algorithm. Unfortunately, none of the five famous identification software could pass the most stringent Null-Test. PatternLab had good performance in both Null-Test and routine search by making a good control on the overfitting with sound design. The fuzzy logics based method presented as another candidate strategy could pass the Null-Test and has competitive efficiency in peptide identification. Filtering the results by appropriate FDR would increase the number of discoveries in an experiment, at the cost of losing control of Type I errors. Thus, it is necessary to utilize some more stringent criteria when someone wants to design or analyze an algorithm/software. The more stringent criteria will facilitate the discovery of latent bugs, faultiness, errors etc. in the algorithm/software. It would be recommended to utilize independent search combining random database with statistics theorem to estimate the accurate FDR of the identified results.

In the past decades, considerable effort has been devoted to developing a sensitive algorithm for peptide identification in Shotgun proteomics. However, little attention has been paid to controlling the reliability of the identification algorithm at the design stage. The Null-Test based on random matching can be utilized to check whether the algorithm has a tendency to make a mistake or has potential bugs, faultiness, errors etc. However, it turns out that none of the five famous identification software could pass the most stringent Null-Test in the present study, which should be taken into account seriously. Accordingly, a candidate strategy based on fuzzy logics has been demonstrated the possibility that an identification algorithm can pass the Null-Test. PatternLab shows that earlier control on overfitting is valuable for designing an efficient algorithm.