Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Powering Value-Based Care with Clinician-Led, Patient-Centered Design: The MyHealth Bundles Program.

NEJM catalyst innovations in care delivery·2026
Same author

The Quiet Surgeon: A Qualitative Analysis of the Introverted Experience Throughout a Career in Academic Surgery.

Annals of surgery open : perspectives of surgical history, education, and clinical approaches·2026
Same author

Missed Opportunities in Donation After Circulatory Death Liver Transplantation: A National Analysis of Non-Allocation and Non-Procurement.

Clinical transplantation·2026
Same author

Validity Evidence for STATUS to Assess Resident Tolerance for and Competence in Communicating Medical Ambiguity.

Academic pediatrics·2026
Same author

A novel risk score helps identify which patients will benefit from thoracic aortic aneurysm repair.

Journal of vascular surgery·2026
Same author

Clarifying the Role of Operation Alone for Pancreas Cancer When Multimodal Therapy Is Not Feasible: In Reply to Hue and colleagues.

Journal of the American College of Surgeons·2025

Related Experiment Video

Updated: Aug 23, 2025

A Computerized Functional Skills Assessment and Training Program Targeting Technology Based Everyday Functional Skills
07:31

A Computerized Functional Skills Assessment and Training Program Targeting Technology Based Everyday Functional Skills

Published on: February 13, 2020

7.0K

Impact of Rater Training on Residents Technical Skill Assessments: A Randomized Trial.

Kristen M Jogerst1, Yoon Soo Park2, Roi Anteby3

  • 1Department of General Surgery, Mayo Clinic Arizona, Phoenix, Arizona; Department of Surgery, Massachusetts General Hospital, Boston, Massachusetts.

Journal of Surgical Education
|November 4, 2022
PubMed
Summary

Rater training improved surgical skill assessment by enhancing score discrimination and scale usage. However, consistent agreement among raters remains a challenge, suggesting consensus scoring for high-stakes evaluations.

Keywords:
ACS/APDS Resident Skills Curriculum's Objective Structured Assessment of Technical Skills (OSATS)Rater TrainingStandard SettingTechnical-Skills Ratings

More Related Videos

Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide
09:52

Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide

Published on: January 15, 2017

17.3K
Setup and Execution of the Rapid Cycle Deliberate Practice Death Notification Curriculum
04:36

Setup and Execution of the Rapid Cycle Deliberate Practice Death Notification Curriculum

Published on: August 5, 2020

4.3K

Related Experiment Videos

Last Updated: Aug 23, 2025

A Computerized Functional Skills Assessment and Training Program Targeting Technology Based Everyday Functional Skills
07:31

A Computerized Functional Skills Assessment and Training Program Targeting Technology Based Everyday Functional Skills

Published on: February 13, 2020

7.0K
Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide
09:52

Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide

Published on: January 15, 2017

17.3K
Setup and Execution of the Rapid Cycle Deliberate Practice Death Notification Curriculum
04:36

Setup and Execution of the Rapid Cycle Deliberate Practice Death Notification Curriculum

Published on: August 5, 2020

4.3K

Area of Science:

  • Medical Education
  • Surgical Skills Assessment
  • Surgical Training

Background:

  • The Objective Structured Assessment of Technical Skills (OSATS) uses checklists and global rating scales (GRS) for evaluating surgical residents.
  • Prior research indicated a need for standardized rater training to improve the reliability of OSATS.
  • This study focuses on handsewn bowel anastomosis (HBA) and vascular anastomosis (VA) as key surgical procedures.

Purpose of the Study:

  • To evaluate the impact of a rater-training curriculum on the discrimination, consistency, and validity of OSATS scoring.
  • To assess whether rater training improves the ability of faculty to differentiate performance levels in surgical tasks.
  • To compare the effectiveness of trained versus untrained raters against a consensus "Gold Score" for HBA and VA.

Main Methods:

  • Development of a rater training video model including GRS orientation and performance anchoring.
  • Randomization of faculty raters into training and control groups.
  • Scoring of resident HBA/VA videos by both groups and comparison to a consensus Gold Score derived via a modified Delphi process.

Main Results:

  • Rater training significantly improved score discrimination and reduced scale shrinkage for both VA and HBA.
  • Trained raters demonstrated better agreement with the Gold Score for VA (κ=0.65) compared to untrained raters (κ=0.57).
  • For HBA, rater training showed a moderate impact on validity, with trained raters (R1 κ=0.71) and untrained raters (R2 κ=0.73) showing similar agreement with the Gold Score.

Conclusions:

  • A rater-training curriculum enhances surgical residents' skill assessment by improving score discrimination and scale utilization.
  • Despite training, persistent inter-rater variability necessitates consensus scoring for high-stakes surgical skill evaluations.
  • Consensus ratings, established through standard-setting processes, offer greater validity than individual faculty assessments in high-stakes scenarios.