Feedback Frequency in Competence by Design: A Quality Improvement Initiative

Otolaryngology-head and neck surgery is in the first wave of residency training programs in Canada to adopt Competence by Design (CBD), a model of competency-based medical education. CBD is built on frequent, low-stakes assessments and requires an increase in the number of feedback interactions. The University of Toronto otolaryngology-head and neck surgery residents piloted the CBD model but were completing only 1 assessment every 4 weeks, which was insufficient to support CBD.

This project aimed to increase assessment completion to once per resident per week using quality improvement methodology.

Stakeholder engagement activities had residents and faculty characterize barriers to assessment completion. Brief electronic assessment forms were completed by faculty on residents' personal mobile devices in face-to-face encounters, and the number completed per resident was tracked for 10 months during the 2016-2017 pilot year. Response to the intervention was analyzed using statistical process control charts.

The first bundled intervention-a rule set dictating which clinical instance should be assessed, combined with a weekly reminder implemented for 10 weeks-was unsuccessful in increasing the frequency of assessments. The second intervention was a leaderboard, designed on an audit-and-feedback system, which sent weekly comparison e-mails of each resident's completion rate to all residents and the program director. The leaderboard demonstrated significant improvement from baseline over 10 weeks, increasing the assessment completion rate from 0.22 to 2.87 assessments per resident per week.

A resident-designed audit-and-feedback leaderboard system improved the frequency of CBD assessment completion.