Traditional tests and job simulations: minority and majority performance and test validities

Area of Science:

• Industrial-Organizational Psychology
• Human Resources Management
• Psychometrics

Background:

• Traditional paper-and-pencil tests are widely used in job applicant selection.
• Concerns exist regarding potential subgroup mean differences in performance on these traditional tests.
• Simulation-based assessments are proposed as an alternative to mitigate such differences.

Purpose of the Study:

• To compare the performance of minority and majority job applicants on traditional tests versus a telephone simulation.
• To assess the measurement characteristics, including subgroup differences and score variances, of both assessment methods.
• To evaluate the relative validity and subgroup fairness of the telephone simulation compared to traditional tests.

Main Methods:

• Assessed performance of 565 minority and 414 majority job applicants.
• Utilized both traditional paper-and-pencil tests and a telephone simulation measuring similar constructs.
• Employed statistical modeling to analyze measurement characteristics and validity.

Main Results:

• Larger subgroup mean differences were observed on traditional tests compared to the telephone simulation.
• Correlations between traditional tests were lower in the majority sample than the minority sample.
• Score variance on traditional tests was significantly larger for minority candidates than majority applicants.

Conclusions:

• The telephone simulation demonstrated smaller subgroup differences, suggesting greater fairness than traditional tests.
• While the simulation's validity was lower, its reduced subgroup differences align with previous research.
• This study provides evidence supporting the relative validity and potential for improved subgroup fairness of simulation-based assessments.