Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

Area of Science:

• Medical Education
• Surgical Training
• Assessment Science

Background:

• Competency-based assessment requires robust validity evidence for reliable evaluation of surgical skills.
• The contemporary framework for validity encompasses five sources of evidence.
• Procedure-specific operative assessment tools are crucial for evaluating surgical competency.

Purpose of the Study:

• To critically assess the validity evidence supporting procedure-specific operative assessment tools in general surgery.
• To evaluate the methodological rigor and educational utility of these assessment tools.

Main Methods:

• Systematic literature search across 8 databases for relevant studies.
• Assessment of validity evidence using the contemporary framework.
• Evaluation of study quality using the Medical Education Research Study Quality Instrument.
• Assessment of educational utility using the Accreditation Council for Graduate Medical Education framework.

Main Results:

• 28 studies and 23 unique tools were evaluated.
• Validity evidence scores varied widely (3-14 out of 15).
• Methodological quality scores were also variable (8.5-15.5 out of 16.5).
• Limited direct reporting of educational utility criteria was observed.

Conclusions:

• A limited number of procedure-specific operative assessment tools in general surgery possess adequate validity evidence.
• Many existing tools are insufficiently studied for high-stakes summative assessments.
• Development of a more robust library of validated operative assessment tools is essential for competency-based general surgery training.