Examining the Predictive Validity of NIH Peer Review Scores

Area of Science:

• Biomedical Research Funding
• Grant Review Processes
• Scientific Evaluation Metrics

Background:

• Empirical demonstration of the predictive validity of National Institutes of Health (NIH) peer review remains limited.
• A common assumption suggests correlating peer review scores with bibliometric indices could validate predictive accuracy.
• This study investigates the feasibility and validity of using bibliometric data to assess NIH peer review's predictive power.

Purpose of the Study:

• To empirically test the predictive validity of NIH peer review.
• To determine if correlating peer review scores with bibliometric indices is a valid method for assessing predictive validity.
• To analyze the relationship between grant application scores and subsequent publication impact.

Main Methods:

• Utilized a large dataset of NIH grant applications and funded projects.
• Examined the correlation between peer review percentile scores and bibliometric indices (e.g., citation counts) of resulting publications.
• Analyzed the distribution of scores for funded versus unfunded applications and considered post-review negotiations.

Main Results:

• Significant restriction of range was observed in applications selected for funding.
• Funded applications with lower peer review scores were not random or representative.
• Negotiations between NIH institutes and applicants altered projects, meaning initial scores did not reflect final funded work.
• Citation metrics alone are insufficient and inappropriate for measuring scientific impact.

Conclusions:

• Retrospective correlation between NIH peer review scores and bibliometric indices is not a valid test of predictive validity.
• Bibliometric indices alone are inadequate measures of scientific impact and may worsen existing research issues.
• The current NIH peer review and evaluation system requires re-evaluation for true predictive accuracy.