The answer key as a source of error in performance assessment.

Area of Science:

• Medical Education
• Psychometrics

Background:

• Performance-based assessments are crucial for evaluating clinical skills.
• Variability in answer key construction can introduce measurement error.

Purpose of the Study:

• To assess the impact of answer key construction variability on measurement error.
• To compare methods for reducing this error in video-based assessments.

Main Methods:

• A video-based format assessing arteriogram interpretation was used.
• Generalizability theory and aggregate scoring were applied to estimate error variance.

Main Results:

• Scorer variability was less than case variability.
• Several scorers significantly reduced measurement error.

Conclusions:

• Answer key construction error is relatively small.
• This error should not impede the use of performance-based item formats.