Related Experiment Video
Updated: Jul 6, 2026
10:39
Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability.
Herbert W Marsh1, Upali W Jayasinghe, Nigel W Bond
1Department of Education, University of Oxford, United Kingdom. herb.marsh@education.ox.ac.uk
The American Psychologist
|April 2, 2008
Summary
Peer review processes for academic grants lack reliability and can be biased. A new "reader system" demonstrated superior reliability and strategic advantages over traditional peer review methods.
Area of Science:
- Academic integrity
- Research evaluation methodologies
- Psychological research standards
Background:
- Traditional peer review is criticized for lacking reliability, validity, generalizability, and for potential biases.
- Despite extensive literature, empirical research on peer review quality and improvement strategies remains limited.
- The Australian Research Council receives numerous grant proposals across diverse disciplines, reviewed globally.
Purpose of the Study:
- To critically evaluate the reliability and biases in traditional peer review of grant applications.
- To assess the impact of applicant, assessor, and interaction characteristics on review outcomes.
- To introduce and evaluate a novel "reader system" as an alternative to traditional peer review.
Main Methods:
- Utilized multilevel cross-classified models to analyze peer reviews of Australian Research Council grant applications.
- Examined potential biases related to applicant and assessor demographics, institutional affiliation, and experience.
- Compared the reliability and strategic advantages of the proposed reader system against traditional peer review.
Main Results:
- Peer reviews were found to lack reliability.
- A significant systematic bias was identified: inflated, unreliable, and invalid ratings from applicant-nominated assessors.
- The reader system proved substantially more reliable and strategically advantageous for psychology and education grant proposals.
Conclusions:
- Traditional peer review systems exhibit significant reliability issues and biases, particularly concerning self-nominated reviewers.
- The proposed reader system offers a more reliable and effective alternative for evaluating academic grant proposals.
- Further research and implementation of improved peer review systems are crucial for academic integrity and research funding allocation.
Related Concept Videos
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
01:22Bias
Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
01:27Halo Effect
The halo effect is a cognitive bias in which an individual's overall impression influences judgments about their specific traits. This psychological phenomenon leads people to associate positive characteristics with those they perceive as generally good and negative characteristics with those they view as bad. This effect is particularly influential in social perception, professional evaluations, and decision-making processes.The Psychological Basis of the Halo EffectThe halo effect is rooted...
01:15Data Validation
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Key parameters for method validation include:
01:03Data Validation
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast, controlled...