Feasibility of an internet-based global ranking instrument

Single-item global ratings are commonly used at the end of undergraduate clerkships and residency rotations to measure specific competencies and/or to compare the performances of individuals against their peers. We hypothesized that an Internet-based instrument would be feasible to adequately distinguish high- and low-ability residents.

After receiving Institutional Review Board approval, we developed an Internet-based global ranking instrument to rank 42 third-year residents (21 in 2008 and 21 in 2009) in a major university teaching hospital's department of anesthesiology. Evaluators were anesthesia attendings and nonphysicians in 3 tertiary-referral hospitals. Evaluators were asked this ranking question: "When it comes to overall clinical ability, how does this individual compare to all their peers?"

For 2008, 111 evaluators completed the ranking exercise; for 2009, 79 completed it. Residents were rank-ordered using the median of evaluator categorizations and the frequency of ratings per assigned relative performance quintile. Across evaluator groups and study years, the summary evaluation data consistently distinguished the top and bottom resident cohorts.

An Internet-based instrument, using a single-item global ranking, demonstrated feasibility and can be used to differentiate top- and bottom-performing cohorts. Although ranking individuals yields norm-referenced measures of ability, successfully identifying poorly performing residents using online technologies is efficient and will be useful in developing and administering targeted evaluation and remediation programs.