Inferring signs from purposeful samples: The role of context in competency assessment

Medical students' clinical competencies are customarily assessed using convenience samples of performance from real practice. The question is how these convenience samples can be turned into purposeful samples to extrapolate students' overall competency profile at the end of medical school, particularly given the context specificity of clinical performance. In this paper, we will address this issue of inferring signs from samples using insights from the discipline of psychology.

We adapted Smith's theory of predictor validity of universals, occupationals and relationals to the context of clinical competency assessment. Universals are characteristics required by all working individuals and therefore not context dependent. Occupationals refer to characteristics required by certain jobs but not others and therefore are dependent on task-related features of an occupation. Relationals are required in a specific organisational context with habitual ways of working together.

Through seven propositions, we assert that generalising from samples of assessed clinical competencies during clerkships to generic competencies (i.e., signs) is dependent on whether characteristics are universals, occupationals and relationals, with universals most and relationals least generalisable.

When determining what types of ratings to use to evaluate medical student competence, medical education has shown too little nuance in considering the degree to which particular characteristics are likely to be generalisable, approaching the issue in an all-or-none manner. Smith's distinction between universals, occupationals and relationals offers a promising way forward that has implications for assessment, student selection and career choice.