The Impact of Using Mean Versus Mode When Assessing Resident Competency

Area of Science:

• Medical Education
• Graduate Medical Education
• Resident Assessment

Background:

• The Accreditation Council for Graduate Medical Education Milestone Project, established in 2014, aimed to standardize resident assessment and progression.
• Despite recommendations against using milestones for direct competency evaluation, many programs have adapted them for this purpose.

Purpose of the Study:

• To investigate the utility of the mode (most frequent evaluation level) versus the arithmetic mean for summarizing resident performance data.
• To compare the frequency and direction of estimation errors when using mean versus mode for milestone evaluations.

Main Methods:

• Analysis of Family Medicine Milestone evaluations from one program over two academic years.
• Calculation of mean and mode scores across 24 residents, 841 forms, and 5897 measurements.
• Comparison of overestimation and underestimation errors between mean and mode across training years and subcompetencies.

Main Results:

• Estimation errors between mean and mode occurred in 22% of comparisons.
• Overestimation errors (mean > mode) constituted 67% of all errors.
• First-year residents accounted for 55% of estimation errors, indicating significant discrepancies in their evaluations.

Conclusions:

• The current trend of using milestones as stand-alone assessment tools necessitates more appropriate data aggregation methods.
• Frequency distributions and the mode provide a more stable and accurate summary for milestone data, given its nominal or ordinal nature.