Standard setting of script concordance tests using an adapted Nedelsky approach

Area of Science:

• Medical Education Assessment
• Psychometrics
• Standard Setting Methodologies

Background:

• Standard setting assigns meaning to assessment scores, crucial for evaluating achievement.
• Current standard setting for script concordance tests (SCTs) often relies on norm-referenced methods.
• Establishing appropriate standards for SCTs remains a significant challenge in assessment.

Purpose of the Study:

• To develop and pilot a criterion-referenced standard setting approach for SCTs.
• To adapt the Nedelsky method for SCT standard setting.
• To compare failure rates of the adapted Nedelsky approach with norm-referenced and expert-referenced methods.

Main Methods:

• A 45-question SCT was administered to second- and third-year medical students.
• Three standard setting approaches were applied: norm-referenced, expert-referenced, and adapted Nedelsky.
• Feasibility and failure rates were measured for each approach.

Main Results:

• All tested standard setting approaches were feasible.
• The adapted Nedelsky approach required an additional 60 minutes for standard setting.
• Failure rates were similar across methods (Year 2: 8.0-9.8%; Year 3: 2.1-7.6%), with adapted Nedelsky at 8.0% and 3.5% respectively.

Conclusions:

• The adapted Nedelsky approach offers a logically justifiable and logistically simple criterion-referenced method for SCT standard setting.
• This method produced failure rates comparable to existing, less objective approaches.
• The adapted Nedelsky approach provides an intermediate option for standard setting in medical assessments.