A statistical model validating triage for the peer review process: keeping the competitive applications in the review...

Triage of grant application at the National Institutes of Health (NIH) is a process whereby an initial screening of applications by a scientific peer review group eliminates applications that are not competitive for awards. The process of application triage has been limited to those applications submitted to the NIH in response to an RFA (Request for Applications). A hypergeometric model was developed to determine the extent to which five, six, seven, or eight member triage teams or subsets of 12-to-20 member full committees could provide a statistically defensible triage decision. Although the intent of triage is to remove from review those applications that are noncompetitive, the model was weighted in favor of the applicant to minimize the likelihood that highly competitive applications would be eliminated. Within the assumptions and rules developed, it was determined that there was little likelihood that the latter would occur. For example, in the worst case scenario, the greatest probability that a highly competitive application would be knocked out of competition is P < or = 0.014 in the case of a five-member triage subset of a 20-member committee. Using the latter case, the model was tested on a set of 73 applications that were submitted to the National Cancer Institute for action at the February 1993 National Cancer Advisory Board. The model selected for triage required that each application be assigned to five reviewers, that each reviewer be blinded to the review assignments of the other reviewers, and that four noncompetitive votes be registered to triage out an application. Each of 19 applications received four to five noncompetitive votes, and were triaged out of the review process. The remaining 54 applications were then reviewed according to the usual NIH review process. Four of the applications received three noncompetitive triage votes each and were either rated as not recommended for further consideration (NRF, n = 2)) or received priority scores > or = 250 (n = 2) (The smaller the priority score the better the technical merit). Thirteen of the 53 applications received two noncompetitive votes. Of the latter, two were not recommended for further consideration and the remaining 11 received priority scores between in excess of 200. The distribution of competitive applications was such that funding was limited to those applications with priority scores of less than 190. Thus, the data suggest that the conservative model is valid such that the likelihood of eliminating a highly competitive application from consideration for funding is remotely small.(ABSTRACT TRUNCATED AT 400 WORDS)