Much ado about differences: why expert-novice comparisons add little to the validity argument

Area of Science:

• Educational Measurement
• Psychometrics
• Clinical Assessment Validation

Background:

• Assessment score validation often uses known-groups comparisons to assess relationships with other variables.
• These studies hypothesize score differences between groups with distinct characteristics, like training status or disease state.

Purpose of the Study:

• To critically evaluate the utility of known-groups comparison studies in providing validity evidence for assessment scores.
• To identify the limitations and flaws inherent in the known-groups comparison methodology.

Main Methods:

• Analysis of the conceptual underpinnings of known-groups comparison studies.
• Examination of the impact of confounding variables on the interpretation of results.

Main Results:

• Known-groups comparisons suffer from nonspecificity, making conclusions ambiguous.
• Confounding variables present multiple plausible explanations for observed group differences, limiting interpretability.

Conclusions:

• While necessary, accurate discrimination in known-groups comparisons is insufficient to validate assessment scores.
• The inherent flaws, particularly confounding, significantly limit the contribution of known-groups comparisons to the overall validity argument.