Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program.
Donna K Ginther1, Misty L Heggeness2,3
1University of Kansas, 1450 Jayhawk Blvd, Lawrence, KS, 66045, United States.
Research Policy
|July 18, 2020
Summary
Scientific peer review effectively predicts high-quality research and future career independence, outperforming program director discretion. This study validates the peer-review system
Area of Science:
- Biomedical research funding
- Scientific evaluation systems
Background:
- The scientific community questions the efficacy of peer review in predicting research success.
- Federal agencies often use a two-step award process involving peer review and programmatic discretion.
Purpose of the Study:
- To evaluate whether peer-review recommendations or programmatic discretion better predict future scientific research success.
- To assess the effectiveness of the National Institutes of Health (NIH) peer-review system.
Main Methods:
- Analysis of data from the Ruth L. Kirschstein National Research Service Award (NRSA) training program.
- Comparison of predictive accuracy between peer-review scores and program staff/institute leader discretion.
- Evaluation of the suitability of regression discontinuity for analyzing scientific funding.
Main Results:
- The existing peer-review system effectively predicts high-quality science and research independence.
- Peer-review recommendations were more accurate predictors than programmatic discretion.
- Regression discontinuity may not be suitable for all scientific funding models and requires cautious application.
Conclusions:
- The current scientific peer-review process functions as intended, accurately identifying promising research.
- Peer review is a superior predictor of scientific success compared to discretionary judgment.
- Caution is advised when using regression discontinuity for federal scientific award analysis.
