Teaching raters to see: A faculty development innovation for robust assessment in health professions education

Trustworthy simulation-based assessment depends on capable raters-but current approaches to rater training are often resource-intensive, inconsistently effective, and conceptually muddled. In practice, many programs focus on delivering training activities (e.g., frame-of-reference or behavioral observation training) without a clear model of the steps that build reliable, trustworthy assessments.We clarify the pathway to reliable, trustworthy assessment skills by distinguishing rater cognition-the cognitive work of recognizing, interpreting, and judging performance-from rater training, the educational processes intended to develop that capability. Rating is not a mechanical act of applying a rubric; it is a skilled perceptual and interpretive process. As we argue, a rubric does not do the seeing for you. While rubrics externalize expert judgment, raters must be trained to recognize and apply that expertise-learning to distinguish observable data from inference and to "see through the eyes of the rubric." Traditional rater training approaches can be reframed as mechanisms that build this underlying cognitive skill.Building on this conceptual clarification, we introduce a staged learning framework for rater readiness. Rather than treating training as a one-time intervention, we propose that raters must be prepared developmentally: first through foundational deliberate practice, then through situational application, and finally through expert-facilitated calibration and ongoing feedback.We illustrate this model through a pilot of an interactive, technology-enabled, partially asynchronous rater training program for the Advocacy-Inquiry Rubric. This pilot shows how simulation programs can greatly reduce the time required from experts to train raters, while also demonstrating feasibility, strong usability, and promising reliability outcomes.