Development of a test to evaluate residents' knowledge of medical procedures

Knowledge of core medical procedures is required by the American Board of Internal Medicine (ABIM) for certification. Efforts to improve the training of residents in these procedures have been limited by the absence of a validated tool for the assessment of knowledge. In this study we aimed to develop a standardized test of procedural knowledge in 3 medical procedures associated with potentially serious complications.

Placement of an arterial line, central venous catheter, and thoracentesis were selected for test development. Learning objectives and multiple-choice questions were constructed for each topic. Content evidence was evaluated by critical care subspecialists. Item test characteristics were evaluated by administering the test to students, residents and specialty clinicians. Reliability of the 32-item instrument was established through its administration to 192 medical residents in 4 hospitals.

Reliability of the instrument as measured by Cronbach's alpha was 0.79 and its test-retest reliability was 0.82. Median score was 53% on a test comprising elements deemed important by critical care subspecialists. Increasing number of procedures attempted, higher self-reported confidence, and increasing seniority were predictors of overall test scores. Procedural confidence correlated significantly with increasing seniority and experience. Residents performed few procedures.

We have successfully developed a standardized instrument to assess residents' cognitive competency for 3 common procedures. Residents' overall knowledge about procedures is poor. Experiential learning is the dominant source for knowledge improvement, but these experiences are increasingly rare.