Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach

Area of Science:

• Psychometric assessment
• Professionalism evaluation
• Educational psychology

Background:

• Situational judgment tests (SJTs) are widely used in admissions to assess professionalism.
• Open-response SJTs show promise, but construct validity concerning assessor evaluation is underexplored.
• Assessor subjectivity may influence the evaluation of diverse responses in SJTs.

Purpose of the Study:

• To investigate construct-relevant and irrelevant factors impacting assessor evaluations in open-response SJTs.
• To address the gap in understanding the construct validity of open-response SJTs.
• To explore how assessors perceive and evaluate professionalism in test-taker responses.

Main Methods:

• Mixed-methods approach combining quantitative content analysis of 160 responses (Casper SJT) and qualitative think-aloud activities with 23 assessors.
• Study I: Correlational and logistic regression analyses to identify predictors of response scores.
• Study II: Thematic analysis of transcribed think-aloud protocols to observe assessor evaluation processes.

Main Results:

• Construct-relevant factors significantly impacted scores, including demonstration of competencies, engagement with dilemma context, justification depth, consideration of perspectives, and solution creativity.
• Construct-irrelevant factors like response flow and phrasing showed mixed results in influencing scores.
• Assessor evaluations were influenced by both the content and the construction of test-taker responses.

Conclusions:

• This study enhances the construct validity evidence for open-response SJTs.
• Findings confirm that SJTs are valid for measuring professional competencies, considering both response content and structure.
• The research provides insights into factors influencing assessor judgments in SJT evaluations.