Rating intervals: an experiment in peer review

The National Institute of Health (NIH) peer review process for research grant applications is one of the largest and most respected systems of its kind in the world. Recently, however, the distribution of raw priority scores voted by NIH study sections has been skewed, and the rating behavior of individual review groups has been quite variable. These phenomena have made funding decisions more difficult. To achieve greater uniformity of rating behavior and a broader description of scores, an experiment was conducted involving 24 study sections. Standard adjectival descriptors and standard rating scales were used. On a random basis, half of the study sections were instructed to vote in units of 0.1 while the other half used an interval of 0.5. The results of this study have now been translated into standard practice at NIH.