Multiple, speeded assessments under scrutiny: Underlying theory, design considerations, reliability, and validity

Area of Science:

• Organizational Psychology
• Human Resources Management
• Assessment Psychology

Background:

• Speeded assessments are increasingly used in employee selection.
• There is a lack of conceptual and empirical evidence supporting their reliability and validity.
• This study addresses the effectiveness of these rapid assessment methods.

Purpose of the Study:

• To conceptualize multiple speeded behavioral job simulations.
• To investigate the reliability and validity of these assessments for predicting future performance.
• To examine the impact of rating aggregation and assessor training on validity.

Main Methods:

• Conceptualization using stimulus and response domain sampling.
• Application of the thin slices of behavior paradigm.
• Two studies involving 96 MBA students assessed in 18 speeded role-plays (3-min each).
• Analysis of aggregated ratings across multiple assessors and role-plays.

Main Results:

• Individual speeded role-plays lacked reliability and validity.
• Aggregated scores across all role-plays and assessors showed high predictive validity (.54).
• Validity remained high even with shorter assessment durations (1 min) and basic assessor training.

Conclusions:

• Aggregating ratings from multiple, diverse situations is crucial for comprehensive domain coverage.
• This aggregation captures both ability and personality traits (e.g., extraversion, agreeableness).
• The findings underscore the importance of domain sampling and caution against using single speeded assessments.