Related Experiment Videos
Examining the effects of differential item (functioning and differential) test functioning on selection decisions:
Stephen Stark1, Oleksandr S Chernyshenko, Fritz Drasgow
1Department of Psychology, University of South Florida, USA.
The Journal of Applied Psychology
|May 27, 2004
Summary
Differential item functioning (DTP) analysis in personnel selection can be hard to interpret. New effect size measures show that despite statistically significant findings, the practical impact on selection decisions is often negligible.
Area of Science:
- Psychometrics
- Educational Measurement
- Personnel Selection
Background:
- Differential item functioning (DTP) is crucial in personnel selection.
- Interpreting DTP results can be challenging due to statistical significance not equating to practical importance.
- Existing methods often lack clear effect size indicators.
Purpose of the Study:
- To propose and evaluate new effect size measures for DTP.
- To assess the practical significance of DTP in various selection contexts.
- To improve the interpretability of DTP findings.
Main Methods:
- Developed two novel effect size measures for DTP.
- Related DTP to mean raw score differences across groups.
- Linked DTP to the 4/5th rule for adverse impact at successive cut scores.
- Examined DTP effects in personality assessment, professional licensure, and college admissions.
Main Results:
- Many items showed bias in large sample analyses.
- The proposed effect size measures were applied to assess practical impact.
- The net magnitude of DTP effects on selection decisions was found to be negligible.
- Despite statistical significance, practical importance was minimal.
Conclusions:
- The proposed effect size measures enhance the interpretation of DTP.
- DTP bias, while present, often has a nugatory practical effect on selection outcomes.
- These findings aid in making more informed decisions in personnel selection and admissions.