Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric...

Area of Science:

• Medical Education Assessment
• Psychometric Evaluation
• Health Professional Competence

Background:

• Accurate assessment of health professionals' competence is vital for public health and quality of care.
• Computerized Adaptive Testing (CAT) using Item Response Theory (IRT) offers improved accuracy and reduced respondent burden.
• There is a growing need for concise and valid evaluations in national medical education.

Purpose of the Study:

• To develop and validate a CAT system for assessing health professional competence.
• To evaluate the psychometric properties of a CAT item bank for medical undergraduates.
• To determine the reliability and validity of the developed CAT system.

Main Methods:

• Developed a CAT item bank from 300 multiple-choice items of the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU).
• Analyzed response data from 2000 Chinese clinical medicine undergraduates.
• Conducted psychometric simulations using both simulated and real response data to evaluate item properties and CAT performance.

Main Results:

• The final CAT item bank comprised 121 items, with parameters estimated using a two-parameter logistic model (2PLM).
• CAT simulations showed sufficient marginal reliability (coefficient > 0.750).
• Demonstrated strong criterion-related validity, with CAT scores correlating highly with SCTCMU aggregate scores (Pearson's r > 0.850).

Conclusions:

• The developed CAT system provides a reliable and valid method for assessing health professional competence.
• The CAT offers a more efficient assessment, leading to shorter test durations and reduced participant burden.
• This psychometric approach supports concise yet effective evaluations in national medical education.