Evaluating Resident Procedural Skills: Faculty Assess a Scoring Tool

Area of Science:

• Medical Education
• Residency Training
• Procedural Skills Assessment

Background:

• Procedural skills assessment is vital for residency training.
• The Council of Academic Family Medicine endorses the Procedural Competency Assessment Tool (PCAT) for family medicine residents.
• The reliability and limitations of the PCAT require evaluation.

Purpose of the Study:

• To evaluate the reliability of the PCAT.
• To identify strengths and limitations of the PCAT in assessing procedural competence.

Main Methods:

• A mixed-methods study involving 18 faculty members assessing a simulated shave biopsy.
• Faculty used the original PCAT, participated in focus groups, and then used a modified PCAT with objective criteria.
• Qualitative analysis explored perceived benefits and challenges of the PCAT.

Main Results:

• Competency ratings dropped from 40% to 21.4% with the modified PCAT (P=.035).
• Respondents rated physicians as competent despite scoring procedure components as novice.
• Qualitative data indicated the PCAT is subjective, leading to wide interpretation variability.

Conclusions:

• Further research on PCAT validity and reliability is necessary.
• The PCAT may need additional objective criteria for improved reliability.
• Residencies could train faculty on PCAT use or adopt more intuitive checklist tools.