The impact of differential item functioning on ability estimation using the Korean Medical Licensing Examination with...

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing

Background:

• Differential item functioning (DIF) can introduce bias in ability estimation.
• Computerized adaptive testing (CAT) environments require robust methods to ensure accurate measurement.
• Real-world data from high-stakes examinations like the Korean Medical Licensing Examination (KMLE) are crucial for validating psychometric models.

Purpose of the Study:

• To investigate the impact of excluding gender-based DIF items on ability estimation accuracy within a CAT environment.
• To evaluate whether removing DIF items improves fairness and precision in ability estimation, particularly for examinees at the extremes of the ability scale.
• To provide practical evidence for fairness-oriented test design in high-stakes CAT.

Main Methods:

• Differential item functioning (DIF) was detected using the residual-based DIF method on data from 3,259 examinees.
• Two CAT conditions were simulated: one with all items and one excluding DIF items.
• Ability estimates were compared against a 'true θ' using bias, root mean square error (RMSE), and correlation.

Main Results:

• Excluding DIF items improved accuracy, reducing RMSE and increasing correlation with true ability.
• DIF removal reduced underestimation for females but increased it for males, leading to fairer estimates across genders.
• Estimation errors were more pronounced at the low and high ends of the ability scale when DIF items were included.

Conclusions:

• Managing DIF in CAT is essential for enhancing fairness and precision in high-stakes testing.
• This study demonstrates the practical implications of DIF for CAT-based measurement using real examinee data.
• Implementing fairness-oriented test design principles can improve the quality of standardized assessments.