Examining the generalizability of ratings across clerkships using a clinical evaluation form

It is necessary to average multiple clinical evaluation form (CEF) observations to obtain a reliable score. Combining CEF observations across clerkships will provide more observations per student, but it is unknown how an across-clerkship generalization affects reliability. The authors conducted generalizability studies on balanced stratified random samples to examine the impact of averaging across clerkships. The study detected a student by clerkship interaction. Although the interaction magnitude was small, it had a large negative impact on the reliability of the mean score. The authors conclude that averaging across clerkships for the purpose of evaluating global clinical skills will produce a less reliable measure for a fixed number of observations than averages calculated within a clerkship for evaluating clerkship specific skills. The content specificity of clinical skills as measured by the CEF may resemble that found using other measurement methods.