Assessing Differential Statement Functioning in Polytomous Multidimensional Pairwise Comparison Items.

Area of Science:

• Educational Psychology
• Psychometrics

Background:

• Multidimensional pairwise comparison (MPC) items are prevalent in educational settings for assessing career interest, values, and personality, aiming to mitigate response biases.
• Differential statement functioning (DSF) occurs when an MPC item statement exhibits varying utility across different demographic groups, a phenomenon under-researched.
• Existing methods for detecting DSF in MPC items are limited, necessitating the development of robust analytical approaches.

Purpose of the Study:

• To adapt and evaluate three methods—equal-mean-utility (EMU), all-other-statement (AOS), and constant-statement (CS)—for detecting differential statement functioning (DSF) in polytomous multidimensional pairwise comparison (MPC) items.
• To assess the performance of these methods using a simulation study, focusing on parameter recovery and DSF detection accuracy.
• To provide an empirical application of the proposed methods in the context of career interest assessment.

Main Methods:

• The study adapted three DSF detection methods (EMU, AOS, CS) for polytomous MPC items, grounded in a Rasch model framework.
• A simulation study was employed to evaluate the accuracy of parameter recovery and the effectiveness of the proposed DSF detection methods under various conditions.
• The constant-statement (CS) method was specifically investigated, utilizing DSF-free statements as anchors.

Main Results:

• Simulation results indicated that the constant-statement (CS) method demonstrates superior performance in accurately estimating parameters and detecting differential statement functioning (DSF) when DSF statements are present.
• The CS method, particularly when employing one or more DSF-free statements as anchors, yielded the most accurate results in the simulation study.
• The study successfully applied the developed methods to an empirical dataset concerning career interest assessment, validating their practical utility.

Conclusions:

• The constant-statement (CS) method is recommended for detecting differential statement functioning (DSF) in polytomous multidimensional pairwise comparison (MPC) items due to its accuracy and robust performance.
• The findings underscore the importance of assessing DSF in MPC items to ensure measurement invariance across different groups in educational and psychological assessments.
• The adapted Rasch model-based methods, especially the CS approach, provide valuable tools for researchers and practitioners evaluating career interest, personality, and values.