Assessing measurement equivalence across rating sources: a multitrait-multirater approach

Area of Science:

• Organizational Psychology
• Human Resource Management

Background:

• Multisource performance ratings are influenced by both the rater (source) and the performance aspect being evaluated (dimension).
• Previous research has explored the impact of rating sources and performance dimensions separately, and the equivalence of ratings across sources.

Purpose of the Study:

• To investigate the measurement equivalence of multisource performance ratings considering both dimension and source effects.
• To analyze the independent and interactive impacts of performance dimensions and rating sources on performance evaluations.

Main Methods:

• Examined the structure of multisource performance ratings.
• Assessed the impact of performance dimensions and rating sources on ratings.
• Investigated the degree of measurement equivalence across different rating sources.

Main Results:

• The influence of the underlying performance dimension on ratings is consistent across all rating sources.
• The impact of the rating source is substantial, nearly as significant as the performance dimension.
• The effect of the rating source varies considerably depending on the specific source providing the rating.

Conclusions:

• Performance ratings are a function of both what is rated and who is doing the rating.
• Rating source significantly influences performance evaluations, and this influence is not uniform across all sources.
• Understanding source-specific effects is crucial for accurate and equitable performance appraisal systems.