"Should We Consider Equity?" A Critical Discourse Analysis of Assessment Experts in an OSCE Design Experiment

Achieving equity in assessment remains a persistent challenge in medical education. While research has focused predominantly on psychometric approaches to fairness, exploration has been limited into how racial equity is embedded at foundational stages of assessment development. In particular, the influence of test developers' racial composition on assessment design has been largely overlooked. This study investigated how the racial makeup of test development teams impacts the incorporation of justice-oriented principles into assessment design. Guided by critical theory and a justice-oriented assessment framework, we employed Foucauldian Critical Discourse Analysis to examine the design processes of three groups of assessment developers (N = 16) purposefully sampled from the US academic medical education community. We grouped participants by racial identity: Group A (white-identifying, n = 5), Group B (non-majority-identifying, n = 5), and Group C (mixed racial composition, n = 6). Following an interactive session on equitable assessment principles, groups collaboratively designed a mock OSCE station assessing communication skills. The data collected included transcribed group discussions, the OSCE station, and participant group reflections on their experiences. Our findings reveal a pervasive gap between equity awareness and equity action. Despite demonstrating clear conceptual understanding of equity principles, participant groups struggled to operationalize these principles in practice. We identified four themes from the discourse analysis: an absence of explicit equity discussions despite the study's stated focus; the sanitization of assessment and weaponization of "culture" as a dividing practice; the treatment of learners as erased, ghostlike figures-homogenous and without agency-while patients were centered as diverse and significant; and the framing of equity as a weapon or curveball rather than a foundational design element. Participant groups frequently defaulted to traditional standardized approaches, expressing discomfort about incorporating equity without introducing stereotypes. Power dynamics and professional norms emerged as significant barriers, with participants deferring to clinical expertise and avoiding confrontation on equity-related issues. Notably, racial diversity among test developers alone did not substantially advance equity in the designed assessment. This study underscores that the test developer's equity awareness and racially diverse representation, while necessary, are insufficient for achieving justice-oriented assessment design. Without intentional structural changes and cultural shifts in how assessment is conceptualized, equity will remain peripheral rather than foundational. These findings offer actionable insights for national licensing bodies, medical schools, and residency programs seeking to embed equity more meaningfully into high-stakes assessment practices, suggesting that frameworks such as Design Justice may provide pathways for inclusive, community-centered approaches to assessment development.