Introducing a model for optimal design of sequential objective structured clinical examinations

Area of Science:

• Medical Education
• Assessment Methodology
• Health Professions Education

Background:

• Sequential Objective Structured Clinical Examinations (OSCE) offer potential cost reductions by screening candidates.
• Effective and accurate design of screening OSCEs is crucial for reliable assessment.
• Previous models lacked comprehensive evaluation of screening test parameters.

Purpose of the Study:

• To develop and evaluate a model for designing and assessing sequential OSCE screening tests.
• To identify optimal parameters for screening OSCEs to ensure accuracy and efficiency.
• To determine factors influencing the success of screening OSCEs in predicting full OSCE outcomes.

Main Methods:

• Developed a model for designing and evaluating screening OSCEs using two pre-internship OSCE datasets.
• Proposed hypothetical screening tests varying in number of stations, selection criteria, and cut-off scores.
• Calculated positive predictive value (PPV), pass rate, and negative predictive value (NPV); employed logistic regression analysis.

Main Results:

• Half of the screening tests achieved 100% PPV (no false positives).
• Most screening OSCEs demonstrated acceptable pass rates and NPV.
• Twenty screening OSCE designs met the "desirable" composite outcome criteria (PPV=100%, pass rate ≥50%, NPV ≥25%).

Conclusions:

• Station selection criteria and cut-off scores significantly impact the accuracy of sequential OSCE screening.
• Careful selection of high-discrimination stations and stringent cut-off scores are recommended for efficient screening OSCEs.
• The developed model provides a framework for creating accurate and cost-effective sequential OSCE designs.