Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having...

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• Standard setting for written examinations is crucial for determining competency.
• The influence of question format and answer key access on pass mark decisions requires investigation.

Purpose of the Study:

• To determine if question format (single best answer question vs. very short answer question) and access to correct answers affect the pass mark set by standard-setters.
• To analyze the impact of these variables on the perceived difficulty of examination items.

Main Methods:

• Trained educators utilized the Angoff method for standard setting on two 50-item tests with identical vignettes.
• One test was in a single best answer question (SBAQ) format, the other in a very short answer question (VSAQ) format.
• Participants were divided into two groups: one with access to correct answers, the other without.

Main Results:

• Standard setters set a significantly lower pass mark for VSAQs compared to SBAQs when they had access to the answers (p=0.002).
• Pass marks for SBAQs were comparable whether or not standard setters had access to the answers.
• A lower pass mark was set for VSAQs when participants had access to the correct answers (p=0.014).

Conclusions:

• Standard setters acknowledge the increased difficulty of VSAQs relative to SBAQs when provided with answer keys.
• This suggests that the format of assessment questions influences the rigor of standard-setting processes.