Post-hoc simulation study of computerized adaptive testing for the Korean Medical Licensing Examination

Area of Science:

• Psychometrics
• Educational Measurement
• Medical Education Assessment

Background:

• Computerized adaptive testing (CAT) is increasingly adopted for licensing examinations due to demonstrated improvements in test efficiency and accuracy.
• The Korean Medical Licensing Examination (KMLE) is a critical assessment for medical professionals in Korea.
• Optimizing CAT algorithms is essential for maintaining the validity and reliability of high-stakes examinations like the KMLE.

Purpose of the Study:

• To investigate and compare various CAT scoring and item selection methods using a simulation approach.
• To evaluate the performance of different statistical models (Rasch, 2-parameter logistic, 3-parameter logistic) for CAT in the KMLE context.
• To identify optimal scoring and item selection strategies for the KMLE to enhance testing accuracy and efficiency.

Main Methods:

• A post-hoc simulation design utilizing real data from the January 2017 KMLE item bank.
• Implementation of CAT algorithms using the 'catR' package in the R programming environment.
• Evaluation of scoring methods including 'modal a posteriori', 'expected a posterior', maximum likelihood estimation, and weighted likelihood estimation.
• Assessment of item selection methods such as maximum posterior weighted information and minimum expected posterior variance.

Main Results:

• The Rasch and 2-parametric logistic (PL) models demonstrated superior accuracy compared to the 3-parameter logistic (3PL) model.
• Posteriori estimation methods ('modal a posteriori' and 'expected a posterior') yielded more accurate ability estimates than maximum likelihood or weighted likelihood estimation.
• Item selection methods prioritizing maximum posterior weighted information and minimum expected posterior variance outperformed other strategies.
• The Rasch model is recommended for improved testing efficiency, potentially reducing overall test length.

Conclusions:

• Simulation studies are crucial for evaluating CAT performance under diverse test conditions before live implementation.
• Predetermined scoring and item selection methods, based on robust simulation findings, are recommended for the KMLE.
• The study provides evidence-based recommendations for optimizing CAT for the KMLE, enhancing assessment quality.