Is a Single-Item Operative Performance Rating Sufficient?

Area of Science:

• Medical Education
• Surgical Training
• Performance Assessment

Background:

• Accurate assessment of resident operative performance is crucial for surgical training.
• Existing operative performance rating (OPR) systems can be time-consuming for supervising surgeons.
• A need exists for an efficient OPR system that encourages frequent completion and provides valid feedback.

Purpose of the Study:

• To determine if a single-item overall OPR score offers a valid and stable appraisal of resident operative performance.
• To assess the feasibility and accuracy of a simplified OPR system in a general surgery residency program.

Main Methods:

• Retrospective comparison of single-item OPR scores with a gold-standard rating system.
• Analysis of 1033 operative performances across various procedures in a general surgery residency program (2001-2012).
• Statistical analysis including correlation and generalizability to assess rating validity and reliability.

Main Results:

• Single-item OPR scores demonstrated high correlation (average 0.95) with gold-standard ratings, with minimal differences (average 0.02 points).
• Misclassifications were infrequent (4.4%) and minor, with most single-item ratings being lower than gold-standard.
• Reliability analyses indicated that 5 observations were needed for 0.80 reliability and 11 for 0.90 reliability.

Conclusions:

• A single-item OPR scale provides a valid and stable assessment comparable to multi-item gold-standard ratings.
• The simplified OPR system is efficient, encouraging more frequent ratings and potentially improving resident feedback.
• While efficient, supplementary qualitative feedback remains essential for detailed resident improvement.