Validity: on meaningful interpretation of assessment data

Area of Science:

• Medical Education Assessment
• Educational Psychology
• Psychometrics

Background:

• Meaningful interpretation of medical education assessments hinges on validity evidence.
• Contemporary understanding defines all validity as construct validity, encompassing multiple facets and evidence sources.
• The Standards for Educational and Psychological Testing identify five key sources for validity evidence: content, response process, internal structure, relationships with other variables, and consequences.

Purpose of the Study:

• To elucidate the concept of construct validity within medical education.
• To illustrate typical sources of validity evidence using examples of written and performance-based examinations.
• To underscore the necessity of robust validity evidence for score interpretation in medical training.

Main Methods:

• Discussion of construct validity principles in medical education.
• Exemplification of validity evidence sources using written examinations.
• Exemplification of validity evidence sources using performance examinations.

Main Results:

• Assessments are not inherently valid or invalid; rather, their scores possess varying degrees of evidence supporting specific interpretations.
• Validity is approached as a hypothesis, utilizing theory, logic, and scientific methods to gather data.
• Arguments for and against specific interpretations of assessment data are constructed from collected evidence.

Conclusions:

• All assessments necessitate evidence supporting the reasonableness of their proposed interpretations, as test data lacks intrinsic meaning.
• The constructs measured by assessments are crucial for students, faculty, administrators, patients, and society, demanding strong scientific evidence of their meaning.
• Validity is an ongoing process, requiring continuous collection and evaluation of evidence to support score interpretations.