F-type testlets and the effects of feedback and case-specificity

Area of Science:

• Medical Education
• Psychometrics
• Assessment Design

Background:

• Introduced "f-type" testlets on the United States Medical Licensing Examination.
• These testlets feature multiple questions linked to a common clinical scenario.
• Examinees may receive indirect feedback on their responses within a testlet.

Purpose of the Study:

• Investigate the impact of f-type testlets on examinee behavior.
• Analyze the effect of performance feedback on examinee responses.
• Examine dependencies between questions within a testlet (case-specificity).

Main Methods:

• Utilized an item response model to predict examinee behavior.
• Compared observed responses to model expectations for f-type testlets.
• Analyzed model-data discrepancies across examinee groups to assess case-specificity and feedback impact.

Main Results:

• Case-specificity effects were observed in most examinee subgroups.
• Examinees who responded unsuccessfully to the initial item and received feedback showed no case-specificity.
• Negative case-specificity effects occurred for those unsuccessful without feedback; positive effects for those successful on the initial item.

Conclusions:

• Responses within f-type testlets are not independent, even after accounting for proficiency and item characteristics.
• Feedback mitigates case-specificity effects for examinees who initially responded unsuccessfully.
• Further research using advanced item response models is recommended to capture these dependencies.