Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Quantifying and Rejecting Outliers: The Grubbs Test01:02

Quantifying and Rejecting Outliers: The Grubbs Test

1.5K
Sometimes, a data set can have a recorded numerical observation that greatly  deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier.  To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.5K
Detection of Gross Error: The Q Test01:00

Detection of Gross Error: The Q Test

6.0K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.0K
Regression Toward the Mean01:52

Regression Toward the Mean

6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
Measures of Central Tendency02:16

Measures of Central Tendency

16.0K
The "center" of a data set is also a way of describing location. The two most widely used measures of the "center" of the data are the mean (average) and the median. The words "mean" and "average" are often used interchangeably. The substitution of one word for the other is common practice. The technical term is "arithmetic mean" and "average" is technically a center location. However, in practice among non-statisticians,...
16.0K
Friedman Two-way Analysis of Variance by Ranks01:21

Friedman Two-way Analysis of Variance by Ranks

180
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
180
What Are Outliers?01:12

What Are Outliers?

3.8K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
3.8K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Workplace-Based Assessment in Orthopaedic Training: A Commentary on Evolution and Growing Pains.

JB & JS open access·2026
Same author

Identifying barriers and facilitators to use of the Canadian emergency department best practices checklist for skin and soft tissue infections.

CJEM·2026
Same author

Do outlier assessors provide useful narrative comments?

Medical teacher·2025
Same author

Beyond Individualism and Independence: Exploring Collectivism and Interdependence as Paradigms of Healthcare and Health Professions Education.

Perspectives on medical education·2025
Same author

Two Decades of Medical Education Scholarship: Mapping Collaboration and Thematic Shifts Using Web of Science (2000-2019).

Perspectives on medical education·2025
Same author

Barriers and facilitators to use of the HEARTRISK6 Scale for acute heart failure presentations in the emergency department: a qualitative study.

CJEM·2025

Related Experiment Video

Updated: Jun 21, 2025

Operant Procedures for Assessing Behavioral Flexibility in Rats
08:30

Operant Procedures for Assessing Behavioral Flexibility in Rats

Published on: February 15, 2015

20.8K

The Mean Delta Method: Quantifying Assessor Stringency and Leniency and Identifying Outliers in Workplace-Based

Sebastian Dewhirst, Warren J Cheung, Timothy Wood

    Academic Medicine : Journal of the Association of American Medical Colleges
    |July 10, 2024
    PubMed
    Summary

    A new mean delta method quantifies assessor stringency and leniency (ASL) in workplace assessments. This method identifies outlier assessors and measures the impact of ASL on trainee scores, aiding targeted feedback.

    More Related Videos

    Testing Tactile Masking between the Forearms
    08:05

    Testing Tactile Masking between the Forearms

    Published on: February 10, 2016

    6.4K
    Three Laboratory Procedures for Assessing Different Manifestations of Impulsivity in Rats
    09:12

    Three Laboratory Procedures for Assessing Different Manifestations of Impulsivity in Rats

    Published on: March 17, 2019

    9.4K

    Related Experiment Videos

    Last Updated: Jun 21, 2025

    Operant Procedures for Assessing Behavioral Flexibility in Rats
    08:30

    Operant Procedures for Assessing Behavioral Flexibility in Rats

    Published on: February 15, 2015

    20.8K
    Testing Tactile Masking between the Forearms
    08:05

    Testing Tactile Masking between the Forearms

    Published on: February 10, 2016

    6.4K
    Three Laboratory Procedures for Assessing Different Manifestations of Impulsivity in Rats
    09:12

    Three Laboratory Procedures for Assessing Different Manifestations of Impulsivity in Rats

    Published on: March 17, 2019

    9.4K

    Area of Science:

    • Medical Education
    • Assessment and Evaluation
    • Workplace Learning

    Background:

    • Assessor stringency and leniency (ASL) significantly impacts workplace-based assessments.
    • Outlier assessors can disproportionately affect evaluation outcomes.
    • Existing methods lack a way to quantify ASL or identify outlier assessors from assessment data.

    Purpose of the Study:

    • To introduce and validate the mean delta method for quantifying ASL.
    • To identify outlier stringent and lenient assessors using workplace assessment data.
    • To examine the net effect of ASL on learners' assessment scores.

    Main Methods:

    • The mean delta method was developed, comparing assessor scores to trainee mean scores.
    • The method was applied to 3,908 end-of-shift assessments from an academic emergency department.
    • Outlier assessors were identified using 1.5 and 2 standard deviation cutoffs.

    Main Results:

    • The mean delta method successfully quantified ASL and identified outlier assessors.
    • 11 and 3 outlier assessors were identified using 1.5 and 2 standard deviation cutoffs, respectively.
    • ASL affected overall scores by more than the mean difference between training years for nearly 25% of learners.

    Conclusions:

    • The mean delta method is a simple, effective tool for quantifying ASL and identifying outlier assessors.
    • This method can quantify the impact of ASL on individual trainees.
    • Findings support using the mean delta method for targeted assessor coaching and monitoring ASL changes.