Station gender bias in a multiple-station test of clinical skills

To apply differential item functioning (DIF) procedures to investigate station gender bias in multiple-station tests of clinical skills, and to compare these results with those obtained by comparing the station-score distributions of men and women examinees.

The data were from 23 stations used in the selection of seven successive cohorts (1987-1993) of candidates to the Ontario Pre-Internship Program for graduates of foreign medical schools. The stations had been used on at least three occasions, with a minimum sample of about 210 candidates per station. Each station's score was expressed as both a binary score and a continuous score, and DIF was assessed using the Mantel-Haenszel procedure with the binary scores and analysis of covariance with the continuous scores. For each station, DIF effect sizes were calculated and compared with the gender-group mean differences.

Using the binary scores, significant DIF was observed for three stations; using the continuous scores, significant DIF was observed for five stations. Significant gender differences were observed in the scores of nine stations. In eight, these differences favored women. Overall, in more stations the direction of DIF favored the men, while the women demonstrated higher levels of ability.

The results suggest the importance of using a DIF approach for controlling the "ability factor" in studies of this kind: although significant gender differences were observed in the continuous score distributions of nine stations, generally these differences were not indicative of station gender bias.