The reliability of the proposal review process for a summer research fellowship program

The authors evaluated the reliability of the proposal review process used by a faculty-student committee to choose medical student researchers for summer fellowships. A total of 82 proposals were reviewed by the committee over a two-year period (43 in 1987 and 39 in 1988); the proposals were assigned ratings in the spring of each year. The 39 students whose proposals received the highest ratings over the two-year period received fellowships and carried out their projects during ten weeks in the summer of each year. In December of both years, the research fellows, along with a total of 12 students whose proposals had been rejected by the program over the two-year period but who had received support from other programs, presented their findings. These research presentations were also judged and rated, and the ratings of all the presentations were compared with the student committee's original ratings of the research proposals. A significant correlation was found between both years' sets of ratings for the proposals and the research presentations. The "accepted" students generally received higher ratings on their presentations than did the "rejected" students; however, the mean of the ratings received by the "accepted" students for their presentations was not significantly higher than that received by the "rejected" students. Even so, the significant correlation between each year's set of ratings for all the students involved demonstrates the soundness of the review process.