A Method for Investigating "Instructional Familiarity" and Discerning Authentic Learning

Area of Science:

• Medical Education
• Psychometrics
• Assessment Design

Background:

• Traditional medical education assessments rely on item difficulty and discrimination.
• Instructional familiarity (IF) is proposed as a crucial, yet often overlooked, psychometric quality indicator.
• Understanding IF is essential for accurately interpreting test score meaning and quality.

Purpose of the Study:

• To re-evaluate "instructional sensitivity" for criterion-referenced assessments.
• To define and emphasize the significance of "instructional familiarity" in modern medical education.
• To empirically assess the impact of IF on medical school examination measures.

Main Methods:

• A mixed-method, action research design was employed in a medical endocrinology course.
• Instructor-rated instructional familiarity (IF) was collected for each examination item.
• Rasch measurement models and correlation analyses were used to investigate IF effects.

Main Results:

• The proposed methodology effectively distinguishes genuine learning from rote memorization.
• Incorporating IF provides valuable contextual information often missed by traditional psychometric analyses.
• This approach offers a more nuanced understanding of student performance and item quality.

Conclusions:

• The study advocates for the adoption of IF assessment in medical education.
• This straightforward methodology improves the validity of inferences drawn from student learning.
• Integrating IF enhances the overall quality and interpretability of assessment data.