"Should We Consider Equity?" A Critical Discourse Analysis of Assessment Experts in an OSCE Design Experiment

Area of Science:

• Medical Education Research
• Health Equity Studies
• Assessment Design

Background:

• Achieving equity in medical education assessment is a persistent challenge.
• Research has primarily focused on psychometric fairness, overlooking racial equity in assessment development.
• The influence of test developers' racial composition on assessment design remains largely unexplored.

Purpose of the Study:

• To investigate how the racial makeup of test development teams impacts the incorporation of justice-oriented principles into assessment design.
• To examine the discourse and design processes of racially diverse and homogenous test development teams.
• To identify barriers and facilitators to embedding racial equity in assessment development.

Main Methods:

• Foucauldian Critical Discourse Analysis of three purposefully sampled test development groups (N=16) from US academic medical education.
• Groups were categorized by racial identity: white-identifying (n=5), non-majority-identifying (n=5), and mixed racial composition (n=6).
• Collaborative design of a mock Objective Structured Clinical Examination (OSCE) station assessing communication skills after an equity principles session; data included discussions, OSCE station, and reflections.

Main Results:

• A significant gap exists between equity awareness and equity in practice; conceptual understanding did not translate into operationalization.
• Four discourse themes emerged: absence of explicit equity discussions, sanitization of assessment, learners treated as homogenous while patients were centered, and equity framed as a barrier.
• Racial diversity in test development teams alone did not substantially advance equity in the designed assessment; traditional approaches were often defaulted to.

Conclusions:

• Test developer equity awareness and racial diversity are necessary but insufficient for justice-oriented assessment design.
• Power dynamics, professional norms, and discomfort with potential stereotyping hinder equity integration.
• Intentional structural changes and cultural shifts are crucial to embed equity foundationally in assessment, moving beyond peripheral considerations. Frameworks like Design Justice offer potential pathways.