Related Experiment Video
Updated: Jul 19, 2026
04:12
Mixed Reality for Education (MRE) Implementation and Results in Online Classes for Engineering
Published on: June 23, 2023
Evaluating peer reviews. Pilot testing of a grading instrument
I D Feurer1, G J Becker, D Picus
1Journal of Vascular and Interventional Radiology, Nashville, TN.
JAMA
|July 13, 1994
Summary
This study developed a reliable grading instrument for evaluating peer review quality. The tool demonstrated strong interrater reliability and preliminary validity, supporting its use by editors.
Area of Science:
- Medical writing and scientific communication.
- Research integrity and quality assessment.
Background:
- Evaluating the quality of peer reviews is crucial for maintaining scientific rigor.
- Standardized tools are needed to ensure consistent and objective assessment of peer reviews.
Purpose of the Study:
- To assess the reliability and preliminary validity of a novel grading instrument designed for editors.
- To provide editors with a structured method for evaluating peer review quality.
Main Methods:
- A consecutive sample of 53 peer reviews from 23 manuscripts was used.
- Interrater reliability was tested using intraclass correlation coefficient and ANOVA (n=41).
- Preliminary criterion-related validity was assessed using Kendall's coefficient of concordance (n=12).
Main Results:
- The grading instrument demonstrated high interrater reliability (Intraclass Correlation Coefficient = 0.84, P < .001).
- Analysis of variance showed no significant difference in mean scores (P = .46).
- Preliminary criterion-related validity was indicated (Kendall's coefficient of concordance = 0.94, P = 0.038).
Conclusions:
- The developed grading instrument is reliable for evaluating peer review quality.
- Content validation is complete, and further criterion-related validation is recommended.
- The instrument offers a promising tool for editors to enhance peer review assessment.
Related Concept Videos
01:29Self-Evaluation Maintenance Model
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
01:24Sources of Self-Esteem II: Performance Feedback
Self-esteem is intricately tied to our perception of competence and our ability to exert control over our lives. One of the primary sources of this perception is performance feedback — the ongoing evaluation of our actions in terms of success and failure. According to Franks and Marolla (1976), people derive self-worth from experiencing themselves as causal agents, capable of achieving goals and overcoming obstacles. This process nurtures a critical component of self-esteem: self-efficacy,...