Related Experiment Video
Updated: Jul 13, 2025

The Other End of the Leash: An Experimental Test to Analyze How Owners Interact with Their Pet Dogs
Published on: October 13, 2017
Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal
Andrew B Speer1, Angie Y Delacruz2, Lauren J Wegmeyer2
1Department of Management and Entrepreneurship, Kelley School of Business, Indiana University.
Abstract:
Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice. (PsycInfo Database Record (c) 2024 APA, all rights reserved).
More Related Videos
Related Concept Videos
Friedman Two-way Analysis of Variance by Ranks
Spearman's Rank Correlation Test
Spearman's test calculates...
Kendall's Coefficient of Concordance
Self-Report Tests of Personality
Reliability and Validity
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...

