Related Experiment Videos
Predicting mastery level on a large-scale standardized patient test: a comparison of case and instrument score-based
A F De Champlain1, M J Margolis, M K Macmillan
1National Board of Medical Examiners, 3750 Market Street, Philadelphia, PA 19104, USA. adechamplain@mail.nbme.org
Summary
This study explored automated scoring for clinical skills assessments, finding it feasible for accurate mastery estimation in standardized patient tests. This approach offers a cost-effective alternative to expert raters, maintaining decision-making quality.
Area of Science:
- Medical Education
- Psychometrics
- Health Professions Education
Background:
- Traditional clinical skills assessments rely on expert raters, posing logistical and cost challenges for large-scale evaluations.
- Expert rating systems are susceptible to rater error, potentially compromising assessment validity and reliability.
Purpose of the Study:
- To identify weighted score-based models using discriminant analysis for accurate mastery level estimation in standardized patient tests.
- To evaluate the predictive accuracy of classification functions for mastery level in a cross-validation sample.
Main Methods:
- Discriminant analysis was employed to develop weighted score-based models.
- A nationally administered standardized patient test was used for examinee performance data.
- Cross-validation was performed to assess the generalizability of the classification functions.
Main Results:
- Weighted score-based models demonstrated feasibility for automated scoring in clinical skills assessments.
- The developed models accurately estimated examinee mastery levels, comparable to expert ratings.
- Classification functions showed reliable predictive accuracy for mastery in a cross-validation sample.
Conclusions:
- Automated scoring procedures for standardized patient tests can be implemented cost-effectively.
- This approach retains critical aspects of expert rater decision-making processes.
- Findings have significant implications for test development, psychometrics, and cost-benefit analyses in medical education.