Assessing DIF among small samples with separate calibration t and Mantel-Haenszel χ² statistics in the Rasch model

The National Council Licensure Examination (NCLEX) program has evaluated differential item functioning (DIF) using the Mantel-Haenszel (M-H) chi-square statistic. Since a Rasch model is assumed, DIF implies a difference in item difficulty between a reference group, e.g., White applicants, and a focal group, e.g., African-American applicants. The National Council of State Boards of Nursing (NCSBN) is planning to change the statistic used to evaluate DIF on the NCLEX from M-H to the separate calibration t-test (t). In actuality, M-H and t should yield identical results in large samples if the assumptions of the Rasch model hold (Linacre and Wright, 1989, also see Smith, 1996). However, as is true throughout statistics, "how large is large" is undefined, so it is quite possible that systematic differences exist in relatively smaller samples. This paper compares M-H and t in four sets of computer simulations. Three simulations used a ten-item test with nine fair items and one potentially containing DIF. To address instability that may result from a ten-item test, the fourth used a 30-item test with 29 fair items and one potentially containing DIF. Depending upon the simulation, the magnitude of population DIF (0, .5, 1.0, and 1.5 z-score units), the ability difference between the focal and reference group (-1, 0, and 1 z-score units), the focal group size (0, 10, 20, 40, 50, 80, 160, and 1000), and the reference group size (500 and 1000) were varied. The results were that: (a) differences in estimated DIF between the M-H and t statistics are generally small, (b) t tends to estimate lower chance probabilities than M-H with small sample sizes, (c) neither method is likely to detect DIF, especially when it is of slight magnitude in small focal group sizes, and (d) M-H does marginally better than t at detecting DIF but this improvement is also limited to very small focal group sizes.