Low agreement among reviewers evaluating the same NIH grant applications

Area of Science:

• Biomedical research funding
• Grant peer review processes

Background:

• National Institutes of Health (NIH) grant funding is highly competitive.
• Declining success rates necessitate robust evaluation methods.
• Limited research exists on reviewer consistency in grant application assessment.

Purpose of the Study:

• To investigate the consistency of reviewer judgments in the NIH grant peer-review process.
• To analyze both qualitative and quantitative evaluations of identical grant applications by multiple reviewers.

Main Methods:

• Replicated the NIH peer-review process.
• Examined ratings and written critiques from 43 individual reviewers for 25 NIH grant applications.
• Analyzed reviewer agreement on application quality and rating translation.

Main Results:

• No significant agreement was found among reviewers regarding the quality of grant applications.
• Reviewers showed no consensus in translating application strengths and weaknesses into numerical scores.
• Application review outcomes appeared dependent on the assigned reviewer rather than the research merit.

Conclusions:

• The current NIH peer-review system exhibits substantial variability in reviewer evaluations.
• Inconsistencies in reviewer judgments raise concerns about the fairness and reliability of grant funding decisions.
• Further research is needed to improve the objectivity and standardization of scientific peer review.