Related Experiment Video
Updated: May 6, 2026

Problem-Solving Before Instruction PS-I: A Protocol for Assessment and Intervention in Students with Different Abilities
Published on: September 11, 2021
The answer key as a source of error in performance assessment
J Norcini1, R Reshetar, R Lipner
1American Board of Internal Medicine, 3624 Market Street, 19104, Philadelphia, PA, USA, jnorcini@ABIM.org.
Objective:
To determine the impact of variability in answer key construction (i.e., option weighting) on total errors of measurement and to compare several designs for reducing this effect.
Methods:
A video-based format that assesses ability to interpret arteriograms is studied because it reproduces with high fidelity an important task faced by cardiologists. Responses given by highly qualified examinees are used to develop answer keys by the aggregate scoring method, and generalizability theory is applied to estimate the error variance in answer key construction.
Sample:
Two hundred and two examinees who volunteered to participate in the study after taking their certifying examination in cardiovascular diseases.
Results:
RESULTS show less scorer variability than case variability; however, several scorers make a sizeable reduction in measurement error.
Conclusion:
Although there is some error in the answer key construction process, its size is relatively small and it should not be a major impediment to the use of performance-based item formats.
Related Concept Videos
Systematic Error: Methodological and Sampling Errors
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Reliability and Validity
Types of Errors: Detection and Minimization
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
Hindsight Biases
Theory of Attribution II: Kelley's Covariation Theory
Errors In Hypothesis Tests

