Fairness, Consistency and Bias in the RACS General Surgery Fellowship Examination

The General Surgery Fellowship Examination is the final hurdle examination before being eligible for fellowship of the Royal Australasian College of Surgeons. The purpose of this study was to assess the consistency and reliability of this examination, specifically whether there was evidence of systemic gender bias.

This is a retrospective audit of marks awarded in the examination between 2018 and 2022. Examination components were marked separately by two examiners, who assigned a consensus mark after discussion. Candidates' marks were analysed according to their demographics. Inter-observer agreement between examining pairs was analysed, as were marks awarded by individual examiners.

Overall, 811 candidates (548 [67%] men and 263 [33%] women) sat the examination, of whom 520 (64%) passed. The pass rate was consistent over the study period. A total of 5672 pairs of examiners' marks were analysed to determine inter-observer agreement. Inter-observer correlation was 0.89 (95% C.I. 0.88-0.89). There were 110 examiners (92 men and 18 women). No examiner's marks were outside the 95% control limit for strictness or leniency compared with their peers. Pass rates were similar for male and female candidates (343/548 [63%] male vs. 177/263 [67%], p = 0.20). There were 102 marking pairs with one female and one male examiner. Inter-rater agreement between these examiners was 0.86 (95% CI: 0.82-0.91) when examining men and 0.76 (95% CI: 0.76-0.85) for women.

We found the examination to be consistent, fair and reliable, with no evidence of systemic gender bias.