Related Experiment Video
Updated: Apr 19, 2026
06:17
Measuring the Strength of Mice
Published on: June 2, 2013
57.3K
Measuring the effectiveness of scientific gatekeeping
Kyle Siler1, Kirby Lee2, Lisa Bero3
1Department of Strategic Management, Rotman School of Management, University of Toronto, Toronto, ON, Canada M5S 3E6; ksiler@gmail.com.
Summary
Scientific peer review is vital, but elite journals sometimes reject highly cited papers. While peer review adds value, it may miss impactful research, indicating potential gatekeeping flaws.
Area of Science:
- Bibliometrics
- Scientific Publishing
- Research Evaluation
Background:
- Peer review is the cornerstone of scientific research evaluation.
- Anecdotal evidence suggests gatekeeping errors in leading journals, including rejection of seminal work.
- Systematic evidence on peer review effectiveness is limited due to restricted access to rejected manuscripts.
Purpose of the Study:
- To systematically assess the effectiveness of scientific gatekeeping in elite medical journals.
- To analyze the relationship between editorial/peer review appraisals and citation outcomes.
- To identify potential flaws in the peer review process regarding impactful research.
Main Methods:
- Analysis of a dataset comprising 1,008 manuscripts submitted to three elite medical journals.
- Comparison of citation counts for manuscripts based on editor decisions (desk-rejected vs. sent for review) and peer reviewer scores.
- Examination of publication and citation data for both accepted and rejected articles.
Main Results:
- Desk-rejected manuscripts received fewer citations than those sent for peer review.
- Lower peer reviewer scores correlated with fewer citations for published articles.
- A significant number of highly cited manuscripts (top 2%) were rejected, with 12 of 14 being desk-rejected.
- Despite some questionable decisions, peer review generally added value in identifying and promoting quality research.
Conclusions:
- Peer review processes in elite journals may struggle to identify and foster the most impactful research.
- Desk rejection appears to be a significant barrier for highly cited manuscripts.
- While not infallible, peer review demonstrates overall value in scientific manuscript evaluation and promotion.
Related Concept Videos
01:27Halo Effect
763
The halo effect is a cognitive bias in which an individual's overall impression influences judgments about their specific traits. This psychological phenomenon leads people to associate positive characteristics with those they perceive as generally good and negative characteristics with those they view as bad. This effect is particularly influential in social perception, professional evaluations, and decision-making processes.The Psychological Basis of the Halo EffectThe halo effect is rooted...
763
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
582
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
582
00:46Feedback Inhibition
59.1K
Biochemical reactions are occurring constantly in cells, converting starting substances to different products, usually with the help of enzymes that speed the reactions. Without enzymes, it would take far too long for most reactions to occur to be useful to the cell!
59.1K
01:28Antimicrobial Effectiveness
2.1K
The effectiveness of antimicrobial agents depends on various factors influencing their ability to eliminate microbial populations. Larger microbial populations require more time for complete eradication, emphasizing the importance of population size analysis when evaluating antimicrobial efficacy.Microbial resistance to antimicrobial agents varies significantly. Highly resilient microorganisms include endospores, gram-negative bacteria, and non-enveloped viruses, while prions are exceptionally...
2.1K
01:02Quantifying and Rejecting Outliers: The Grubbs Test
5.0K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
5.0K
01:32The Scientific Method
275.9K
The scientific method is a detailed, empirical problem-solving process used by biologists and other scientists. This iterative approach involves formulating a question based on observation, developing a testable potential explanation for the observation (called a hypothesis), making and testing predictions based on the hypothesis, and using the findings to create new hypotheses and predictions.
Generally, predictions are tested using carefully-designed experiments. Based on the outcome of these...
Generally, predictions are tested using carefully-designed experiments. Based on the outcome of these...
275.9K