When raters generalize: Examining sources of halo effects with mixture Rasch facets models.

Area of Science:

• Psychometrics
• Cognitive Psychology
• Statistical Modeling

Background:

• Halo effects are a pervasive cognitive bias in assessments, leading to rating errors.
• Existing measurement models lack the capability to identify the sources and detect halo effects.
• Understanding the psychological mechanisms behind halo effects is crucial for accurate evaluations.

Purpose of the Study:

• Propose a general mixture Rasch facets model for halo effects (MRFM-H).
• Develop two specific models (MRFM-H(GI) and MRFM-H(ID)) based on distinct psychological mechanisms: general impressions and inadequate criterion discrimination.
• Implement and validate these models using Bayesian inference, simulations, and real-world data.

Main Methods:

• Developed a general mixture Rasch facets model for halo effects (MRFM-H).
• Derived two specific models: MRFM-H(GI) for general impressions and MRFM-H(ID) for criterion discrimination.
• Employed Bayesian inference for model implementation and validation through simulation studies and real-data analysis.

Main Results:

• Model classification accuracy for halo-inducing persons depends on the number of raters and criteria.
• Sufficient classification accuracy (90%) requires at least 25 ratings per rater-person combination.
• Ignoring halo effects biases criterion estimates but minimally impacts person and rater estimates.
• Bayesian fit statistics (WAIC, WBIC) effectively identified the correct data-generating model.

Conclusions:

• The proposed MRFM-H models offer a robust framework for detecting and understanding halo effects in assessments.
• The models can differentiate between general impression and criterion discrimination biases.
• These models have practical utility in various assessment contexts to improve rating accuracy and validity.