Myths we tell ourselves about validity: A justice-oriented reckoning in medical education

Medical education has moved beyond debating whether objectivity in assessment is achievable, acknowledging that assessment is inherently subjective. Yet the pursuit of objectivity, through standardisation, bias reduction and statistical rigour, remains a cherished goal. This paper argues that this lingering pursuit of objectivity undermines strong validity arguments and produces assessments that fail to engage meaningfully with the lived realities and possible futures of marginalised learners and patients.

Drawing on justice-oriented assessment validity (JAV) frameworks and disability critical race theory (DisCrit), we examine four pervasive myths in medical education assessment: (1) Constructs are universal and stable, (2) removing bias from assessment is sufficient, (3) flexibility decreases rigour and (4) numbers are neutral. These myths allow educators to make validity arguments that ignore or explain away harm. We demonstrate how each myth, when examined from the standpoint of disabled, racially and ethnically marginalised learners, reveals itself as both inaccurate and harmful.

We argue that validity arguments are not merely technical exercises but moral endeavours that determine whose knowledge counts and who is centred or excluded in defining competence. Rooted in the eugenic origins of educational measurement, contemporary assessment practices continue to privilege dominant group norms while systematically disadvantaging marginalised individuals and groups. A justice-oriented approach, informed by JAV and DisCrit, requires centring marginalised perspectives, embracing flexibility as essential for accurate measurement, critically examining how numbers can obscure inequities and demonstrating that assessments work for those most likely to be harmed by them.