Development and evaluation of the General Surgery Objective Structured Assessment of Technical Skill (GOSATS)

Area of Science:

• Surgical Education
• Medical Simulation
• Competency-Based Assessment

Background:

• Technical skill acquisition is crucial for surgical specialty training.
• Current assessment tools for surgical technical skills at training completion are limited.
• Competency-based training emphasizes the need for reliable skill evaluation.

Purpose of the Study:

• To develop and validate a simulated technical skills examination.
• To assess graduating general surgery trainees (postgraduate year 5).
• To provide a tool for direct technical skills assessment.

Main Methods:

• Developed an eight-station, procedure-based simulated general surgery technical skills examination.
• Utilized blinded, board-certified general surgeons to rate trainee performance.
• Employed Cronbach's alpha for reliability, a conjunctive model for pass score setting, and Subkoviak methodology for pass-fail decision reliability.

Main Results:

• Included 10 PGY3 and 9 PGY5 trainees.
• Achieved interstation reliability of 0.66 and inter-rater reliability ranging from 0.76 to 0.97.
• Set a pass score at 63.1%. PGY5 trainees had a 78% pass rate versus 30% for PGY3 trainees (P=0.037).

Conclusions:

• A summative general surgery technical skills examination was successfully developed.
• The examination demonstrated reliability indices suitable for high-stakes assessments.
• Further evaluation is necessary before implementation in certification decisions.