Is case-specificity content-specificity? An analysis of data from extended-matching questions

Area of Science:

• Medical Education
• Psychometrics
• Cognitive Science

Background:

• Case-specificity, or variability in performance across different cases, is a persistent challenge in medical education assessment.
• Understanding the causes of case-specificity is crucial for improving the validity and reliability of medical assessments.
• The content-specificity hypothesis suggests that variable knowledge levels across topics drive performance variability.

Purpose of the Study:

• To investigate the hypothesis that content knowledge variability is the primary driver of case-specificity in medical education.
• To analyze the sources of variance in trainee performance using generalizability theory.
• To evaluate the impact of test format, specifically extended-matching items (EMIs), on case-specificity.

Main Methods:

• Generalizability analysis was applied to the results of a written test comprising 159 extended-matching items (EMIs).
• The test was administered to two cohorts of general practice trainees (N=227) at a single university.
• Variance components attributed to subjects, topics, items, and their interactions were quantified.

Main Results:

• The variance component for subjects was minimal, indicating limited individual differences in overall performance.
• Variance attributed to topics was smaller than variance attributed to items, contradicting the content-specificity hypothesis.
• The primary source of error variance was the interaction between subjects and items (two-thirds of error), suggesting performance is highly case-dependent.

Conclusions:

• Topical knowledge alone does not adequately explain the observed case-specificity in this study.
• The structure of knowledge and reasoning strategies, such as pattern recognition, may be more influential factors.
• Test format influences the interpretation of case-specificity; increasing topic breadth over item depth enhances reliability but increases testing time.