Constructed response formats and their effects on minority-majority differences and validity

Area of Science:

• Organizational Psychology
• Human Resources Management
• Psychometric Testing

Background:

• Organizations face challenges in selecting immigrants due to potential score disparities between minority (immigrant) and majority (native) groups.
• Traditional selection procedures may inadvertently widen these differences, impacting fairness and predictive accuracy.
• Situational judgment tests (SJTs) are a common selection tool, but their response formats may influence observed group differences.

Purpose of the Study:

• To investigate the impact of different SJT response formats on minority-majority differences in test scores.
• To explore whether more realistic and engaging response formats can reduce score gaps while maintaining validity.
• To examine the underlying mechanisms, such as cognitive load and rater effects, contributing to these differences.

Main Methods:

• Two field experiments were conducted using SJTs with job seekers and incumbents.
• Study 1 compared multiple-choice versus written constructed response formats.
• Study 2 compared written constructed versus audiovisual constructed response formats, analyzing rater effects on anonymized responses.

Main Results:

• Written constructed responses significantly reduced minority-majority differences compared to multiple-choice formats (d = .28 vs. d = .92).
• Audiovisual constructed formats further reduced these differences (d = .09 vs. d = .41) without compromising validity.
• Anonymized rater evaluations of transcribed audiovisual responses showed larger differences, suggesting rater bias reduction in the audiovisual format.

Conclusions:

• Altering SJT response modality to more realistic formats, particularly audiovisual constructed responses, effectively minimizes score disparities between immigrant and native groups.
• These findings suggest that SJTs can be designed to be more equitable and valid selection tools.
• The study highlights the practical implications for organizational selection practices, advocating for the use of innovative response formats.