Validity and Feasibility of the Minicard Direct Observation Tool in 1 Training Program

Availability of reliable, valid, and feasible workplace-based assessment (WBA) tools is important to allow faculty to make important and complex judgments about resident competence. The Minicard is a WBA direct observation tool designed to provide formative feedback while supporting critical competency decisions.

The purpose of this study was to collect validity and feasibility evidence for use of the Minicard for formative assessment of internal medicine residents.

We conducted a retrospective cohort analysis of Minicard observations from 2005-2011 in 1 institution to obtain validity evidence, including content (settings, observation rates, independent raters); response process (rating distributions across the scale and ratings by month in the program); consequences (qualitative assessment of action plans); and feasibility (time to collect observations).

Eighty faculty observers recorded 3715 observations of 73 residents in the inpatient ward (43%), clinic (39%), intensive care (15%), and emergency department (3%) settings. Internal medicine residents averaged 28 (SD=8.4) observations per year from 9 (SD=4.1) independent observers. Minicards had an average of 5 (SD=5.1) discrete recorded observations per card. Rating distributions covered the entire rating scale, and increased significantly over the time in training. Half of the observations included action plans with action-oriented feedback, 11% had observational feedback, 9% had minimal feedback, and 30% had no recorded plan. Observations averaged 15.6 (SD=9.5) minutes.

Validity evidence for the Minicard direct observation tool demonstrates its ability to facilitate identification of "struggling" residents and provide feedback, supporting its use for the formative assessment of internal medicine residents.