The Mean Delta Method: Quantifying Assessor Stringency and Leniency and Identifying Outliers in Workplace-Based...

Area of Science:

• Medical Education
• Assessment and Evaluation
• Workplace Learning

Background:

• Assessor stringency and leniency (ASL) significantly impacts workplace-based assessments.
• Outlier assessors can disproportionately affect evaluation outcomes.
• Existing methods lack a way to quantify ASL or identify outlier assessors from assessment data.

Purpose of the Study:

• To introduce and validate the mean delta method for quantifying ASL.
• To identify outlier stringent and lenient assessors using workplace assessment data.
• To examine the net effect of ASL on learners' assessment scores.

Main Methods:

• The mean delta method was developed, comparing assessor scores to trainee mean scores.
• The method was applied to 3,908 end-of-shift assessments from an academic emergency department.
• Outlier assessors were identified using 1.5 and 2 standard deviation cutoffs.

Main Results:

• The mean delta method successfully quantified ASL and identified outlier assessors.
• 11 and 3 outlier assessors were identified using 1.5 and 2 standard deviation cutoffs, respectively.
• ASL affected overall scores by more than the mean difference between training years for nearly 25% of learners.

Conclusions:

• The mean delta method is a simple, effective tool for quantifying ASL and identifying outlier assessors.
• This method can quantify the impact of ASL on individual trainees.
• Findings support using the mean delta method for targeted assessor coaching and monitoring ASL changes.